Can I use Network Graphs to Expand our MOD Navigator?
Initial Research
Looking to increase the scope of the current project after deploying it yesterday, my first thought is how to increase the overall size of the project, moving from 1 singular chapter, to as big as possible. This has made me look into how viable it is to increase the project to be all MOD documents (within reason).
As there is a number of classified documents present in the MOD, I am unable to put all JTTP, JSP, JDP, etc documents into the MOD Navigator, as I have no access to many documents.
To best visualise the total MOD/NATO documents relating to JTTP 4-05, I would like to create a Network Graph of all the documents relating to the entire JTTP 4-05. From there I can look at all documents that relate to each individual document (where I can, as some are either not present on the internet, or classified. Where they are not present on the internet, but have no reason not to be, I will FOI request a copy where possible).
In JTTP 4-05, there is a number of documents mentioned directly in the 670 page document:
- JSP 375 MOD Health and Safety Handbook
- JSP 434 Defence Construction in the Built Environment
- JSP 435 Defence Estate Management
- JSP 315 (Scale 5, Training Camps and Transit Accommodation)
- JSP 403 Handbook of Defence Land Ranges Safety
- JSP 567 Contractor Support to Operations (CSO), 5th Edition, 2009
- JDP 3-40 Security and Stabilisation: The Military Contribution
- JDP 3-64.1 Force Protection Engineering
- Joint Warfare Publication (JWP) 4-01.1 (water quality standards)
- Allied Joint Publication (AJP) 4-00 Logistics, December 2003
- Allied Joint Publication (AJP)-4.5(A), Host Nation Support Doctrine and Procedures
- NATO Allied Administrative Publication (AAP)-6 NATO Glossary of Terms and Definitions (2012)
- ATP-52(B) NATO Land Force Military Engineer Doctrine
- STANAG 2136, Water Quality Standards
- STANAG 2280, Classification of Field Fortifications and Deployable Protective Structures
- STANAG 2882, Field Sanitation
- STANAG 2885, Emergency Supply of Water
- STANAG 4133, Electrical Power Generation
- QSTAG 1176 (Quadripartite Standardisation Agreement No 1176 Edition 1, Minimum Standards for the Provision of Short Term Camps, 2 December 1997)
- Military Engineering Volume XXI, Military Engineer Support in the Land Environment
- AFM Vol 1, Part 10 Counter Insurgency Operations, January 2010
- AFM 10, Stabilisation Operations
- Def Stan 20-13 Generic Base Architecture
- Health Technical Memoranda (HTM)
- Health Building Notes (HBN)
- Deployable works commercial toolkit (DWCT)
- Army Engineering Support Publication (AESP)
In order to show this graph, I made a repository for this network graph, using a flask backend, a HTML frontend, and local json storage (as I did not see the benefits of connecting this to cloud storage, as it is a simple diagram for research purposes).
Code
Layout:
Doc_Graph/ ├── .git/ ├── .python-version ├── .venv/ ├── README.md ├── app.py ├── graph_data.json ├── main.py ├── pyproject.toml ├── templates/ │ └── index.html └── uv.lock
I used a uv virtual environment as it is lightweight and very quick to set up.
App.py
from flask import Flask, render_template, jsonify, request
import networkx as nx
from networkx.readwrite import json_graph
import json
import os
app = Flask(__name__)
# --- Graph data storage ---
GRAPH_FILE = 'graph_data.json'
def load_graph():
"""Load graph from JSON file if it exists"""
if os.path.exists(GRAPH_FILE):
try:
with open(GRAPH_FILE, 'r') as f:
data = json.load(f)
return json_graph.node_link_graph(data)
except (json.JSONDecodeError, KeyError):
print(f"Warning: Could not load {GRAPH_FILE}, starting with empty graph")
return nx.Graph()
def save_graph(graph):
"""Save graph to JSON file"""
try:
data = json_graph.node_link_data(graph)
with open(GRAPH_FILE, 'w') as f:
json.dump(data, f, indent=2)
except Exception as e:
print(f"Error saving graph: {e}")
# Create the graph object and load existing data
g = load_graph()
@app.route('/')
def index():
# This route just serves the main HTML page
return render_template('index.html')
@app.route('/graph_data')
def graph_data():
# This route provides the current state of the graph as JSON
data = json_graph.node_link_data(g)
return jsonify(data)
@app.route('/add_root_node', methods=['POST'])
def add_root_node():
# This route handles adding root nodes (no connections required)
data = request.get_json()
new_node_id = data.get('id')
if not new_node_id:
return jsonify({'status': 'error', 'message': 'Node ID cannot be empty.'}), 400
if g.has_node(new_node_id):
return jsonify({'status': 'error', 'message': 'Node already exists.'}), 400
# Add the new node without any connections
g.add_node(new_node_id, name=new_node_id)
# Save the graph to file
save_graph(g)
return jsonify({'status': 'success', 'node_id': new_node_id, 'message': 'Root node added successfully'})
@app.route('/add_node', methods=['POST'])
def add_node():
# This route handles adding new nodes or creating relationships between existing nodes
data = request.get_json()
new_node_id = data.get('id')
target_node_id = data.get('target')
if not new_node_id:
return jsonify({'status': 'error', 'message': 'Node ID cannot be empty.'}), 400
# Check if node already exists
node_exists = g.has_node(new_node_id)
if node_exists and target_node_id:
# Node exists and we have a target - create a relationship
if not g.has_node(target_node_id):
return jsonify({'status': 'error', 'message': 'Target node for link does not exist.'}), 400
# Check if edge already exists
if g.has_edge(new_node_id, target_node_id):
return jsonify({'status': 'error', 'message': 'Connection already exists between these nodes.'}), 400
g.add_edge(new_node_id, target_node_id)
save_graph(g)
return jsonify({'status': 'success', 'node_id': new_node_id, 'message': f'Created connection between {new_node_id} and {target_node_id}'})
elif node_exists and not target_node_id:
# Node exists but no target specified
return jsonify({'status': 'error', 'message': 'Node already exists. Specify a target to create a connection.'}), 400
else:
# Node doesn't exist - create it
g.add_node(new_node_id, name=new_node_id)
# If a target node is specified and exists, create an edge
if target_node_id:
if not g.has_node(target_node_id):
return jsonify({'status': 'error', 'message': 'Target node for link does not exist.'}), 400
g.add_edge(new_node_id, target_node_id)
save_graph(g)
message = f'Node {new_node_id} created'
if target_node_id:
message += f' and connected to {target_node_id}'
return jsonify({'status': 'success', 'node_id': new_node_id, 'message': message})
@app.route('/delete_node', methods=['POST'])
def delete_node():
# This route handles deleting nodes and all their connections
data = request.get_json()
node_id = data.get('id')
if not node_id:
return jsonify({'status': 'error', 'message': 'Node ID cannot be empty.'}), 400
if not g.has_node(node_id):
return jsonify({'status': 'error', 'message': 'Node does not exist.'}), 400
# Remove the node (this also removes all connected edges)
g.remove_node(node_id)
# Save the graph to file
save_graph(g)
return jsonify({'status': 'success', 'node_id': node_id, 'message': f'Node {node_id} deleted successfully'})
if __name__ == '__main__':
app.run(debug=True)Index.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>D3 Document Graph</title>
<style>
body { font-family: sans-serif; display: flex; margin: 0; height: 100vh; overflow: hidden; }
.controls {
padding: 10px;
border-right: 1px solid #ccc;
width: 300px;
overflow-y: auto;
background: #f9f9f9;
}
.graph-container {
flex: 1;
position: relative;
overflow: hidden;
}
.links line { stroke: #999; stroke-opacity: 0.6; stroke-width: 2px; }
.nodes circle { stroke: #fff; stroke-width: 2px; }
.nodes circle.root { fill: #ff6b6b; }
.nodes circle.connected { fill: #4ecdc4; }
.nodes circle.leaf { fill: #45b7d1; }
text {
font-size: 12px;
pointer-events: none;
fill: #333;
font-weight: bold;
}
form { display: flex; flex-direction: column; }
input, select, button { margin-bottom: 10px; padding: 8px; border: 1px solid #ddd; border-radius: 4px; }
button { background: #007acc; color: white; cursor: pointer; }
button:hover { background: #005a9e; }
#error-message { margin-top: 10px; padding: 5px; border-radius: 4px; }
.zoom-controls {
position: absolute;
top: 10px;
right: 10px;
z-index: 1000;
}
.zoom-controls button {
margin: 2px;
padding: 8px 12px;
background: rgba(0, 122, 204, 0.8);
color: white;
border: none;
border-radius: 4px;
cursor: pointer;
}
</style>
</head>
<body>
<div class="controls">
<h2>Add Document</h2>
<form id="add-node-form">
<label for="new-node-id">New Document Name:</label>
<input type="text" id="new-node-id" name="id" required>
<label for="target-node-id">Link to Existing Document (optional):</label>
<select id="target-node-id" name="target">
<option value="">No connection (root node)</option>
</select>
<button type="submit">Add Node</button>
</form>
<h2>Delete Document</h2>
<form id="delete-node-form">
<label for="delete-node-id">Select Document to Delete:</label>
<select id="delete-node-id" name="id" required>
<option value="">Select a document to delete</option>
</select>
<button type="submit" style="background: #dc3545;">Delete Node</button>
</form>
<div id="error-message" style="color: red;"></div>
<div id="success-message" style="color: green;"></div>
</div>
<div class="graph-container">
<div class="zoom-controls">
<button onclick="zoomIn()">+</button>
<button onclick="zoomOut()">-</button>
<button onclick="resetZoom()">Reset</button>
</div>
<svg width="100%" height="100%"></svg>
</div>
<script src="https://d3js.org/d3.v5.min.js"></script>
<script>
const svg = d3.select("svg");
const width = window.innerWidth - 300; // Account for controls width
const height = window.innerHeight;
const form = d3.select('#add-node-form');
const deleteForm = d3.select('#delete-node-form');
const targetNodeSelect = d3.select('#target-node-id');
const deleteNodeSelect = d3.select('#delete-node-id');
const errorMessage = d3.select('#error-message');
const successMessage = d3.select('#success-message');
// Set up SVG dimensions
svg.attr("width", width).attr("height", height);
// Create zoom behavior
const zoom = d3.zoom()
.scaleExtent([0.1, 4])
.on("zoom", () => {
g.attr("transform", d3.event.transform);
});
svg.call(zoom);
// Create container for graph elements
const g = svg.append("g");
let link, node, label;
let graphData = { nodes: [], links: [] };
const simulation = d3.forceSimulation()
.force("link", d3.forceLink().id(d => d.id).distance(100))
.force("charge", d3.forceManyBody().strength(-400))
.force("center", d3.forceCenter(width / 2, height / 2));
// Zoom functions
function zoomIn() {
svg.transition().duration(300).call(
zoom.scaleBy, 1.5
);
}
function zoomOut() {
svg.transition().duration(300).call(
zoom.scaleBy, 1 / 1.5
);
}
function resetZoom() {
svg.transition().duration(500).call(
zoom.transform,
d3.zoomIdentity
);
}
function getNodeType(node) {
const connections = graphData.links.filter(link =>
link.source.id === node.id || link.target.id === node.id
);
if (connections.length === 0) return 'root';
if (connections.length === 1) return 'leaf';
return 'connected';
}
function update() {
// Links
link = g.select(".links")
.selectAll("line")
.data(graphData.links, d => `${d.source.id}-${d.target.id}`);
link.exit().remove();
link = link.enter().append("line").merge(link);
// Nodes
node = g.select(".nodes")
.selectAll("g")
.data(graphData.nodes, d => d.id);
node.exit().remove();
const nodeEnter = node.enter().append("g").call(drag(simulation));
nodeEnter.append("circle")
.attr("r", 15)
.attr("class", d => getNodeType(d));
nodeEnter.append("text")
.attr("dy", 5)
.attr("text-anchor", "middle")
.text(d => d.id);
node = nodeEnter.merge(node);
// Update circle colors based on node type
node.select("circle").attr("class", d => getNodeType(d));
// Simulation
simulation.nodes(graphData.nodes).on("tick", ticked);
simulation.force("link").links(graphData.links);
simulation.alpha(1).restart();
// Update dropdown
updateDropdown();
}
function updateDropdown() {
// Store current selections
const currentTargetSelection = targetNodeSelect.property('value');
const currentDeleteSelection = deleteNodeSelect.property('value');
// Clear and rebuild target dropdown
targetNodeSelect.selectAll('option').remove();
targetNodeSelect.append('option')
.attr('value', '')
.text('No connection (root node)');
// Clear and rebuild delete dropdown
deleteNodeSelect.selectAll('option').remove();
deleteNodeSelect.append('option')
.attr('value', '')
.text('Select a document to delete');
graphData.nodes.forEach(node => {
// Add to target dropdown
targetNodeSelect.append('option')
.attr('value', node.id)
.text(node.id);
// Add to delete dropdown
deleteNodeSelect.append('option')
.attr('value', node.id)
.text(node.id);
});
// Restore selections if still valid
if (currentTargetSelection && graphData.nodes.some(n => n.id === currentTargetSelection)) {
targetNodeSelect.property('value', currentTargetSelection);
}
if (currentDeleteSelection && graphData.nodes.some(n => n.id === currentDeleteSelection)) {
deleteNodeSelect.property('value', currentDeleteSelection);
}
}
function ticked() {
link
.attr("x1", d => d.source.x)
.attr("y1", d => d.source.y)
.attr("x2", d => d.target.x)
.attr("y2", d => d.target.y);
node.attr("transform", d => `translate(${d.x},${d.y})`);
}
const drag = simulation => {
function dragstarted(d) {
if (!d3.event.active) simulation.alphaTarget(0.3).restart();
d.fx = d.x; d.fy = d.y;
}
function dragged(d) { d.fx = d3.event.x; d.fy = d3.event.y; }
function dragended(d) {
if (!d3.event.active) simulation.alphaTarget(0);
d.fx = null; d.fy = null;
}
return d3.drag().on("start", dragstarted).on("drag", dragged).on("end", dragended);
}
// --- Form Submission Logic ---
form.on('submit', () => {
d3.event.preventDefault();
errorMessage.text('');
successMessage.text('');
const newId = document.getElementById('new-node-id').value;
const targetId = document.getElementById('target-node-id').value;
const endpoint = targetId ? '/add_node' : '/add_root_node';
const payload = targetId ? { id: newId, target: targetId } : { id: newId };
fetch(endpoint, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(payload),
})
.then(response => response.json())
.then(data => {
if (data.status === 'success') {
successMessage.text(data.message || 'Node added successfully');
return d3.json("/graph_data");
} else {
errorMessage.text('Error: ' + data.message);
throw new Error(data.message);
}
})
.then(graph => {
graphData = graph;
update();
document.getElementById('new-node-id').value = '';
})
.catch(error => {
console.error('Error:', error);
if (!errorMessage.text()) {
errorMessage.text('A network error occurred.');
}
});
});
// --- Delete Form Submission Logic ---
deleteForm.on('submit', () => {
d3.event.preventDefault();
errorMessage.text('');
successMessage.text('');
const nodeIdToDelete = document.getElementById('delete-node-id').value;
if (!nodeIdToDelete) {
errorMessage.text('Please select a document to delete.');
return;
}
if (confirm(`Are you sure you want to delete "${nodeIdToDelete}" and all its connections?`)) {
fetch('/delete_node', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ id: nodeIdToDelete }),
})
.then(response => response.json())
.then(data => {
if (data.status === 'success') {
successMessage.text(data.message || 'Node deleted successfully');
return d3.json("/graph_data");
} else {
errorMessage.text('Error: ' + data.message);
throw new Error(data.message);
}
})
.then(graph => {
graphData = graph;
update();
document.getElementById('delete-node-id').value = '';
})
.catch(error => {
console.error('Error:', error);
if (!errorMessage.text()) {
errorMessage.text('A network error occurred.');
}
});
}
});
// Handle window resize
window.addEventListener('resize', () => {
const newWidth = window.innerWidth - 300;
const newHeight = window.innerHeight;
svg.attr("width", newWidth).attr("height", newHeight);
simulation.force("center", d3.forceCenter(newWidth / 2, newHeight / 2));
simulation.alpha(0.3).restart();
});
// --- Initial Load ---
g.append("g").attr("class", "links");
g.append("g").attr("class", "nodes");
d3.json("/graph_data").then(graph => {
graphData = graph;
update();
});
</script>
</body>
</html>Results
Here is the current Network graph describing this:
I encountered an issue relating to the creation of this network graph, with there inevitably being laws linked to all of these government publications, as all publications and standards follow on from previous UK law standards set. I am going to include them tentatively, but will further confer with Lucien in order to reformat and rescale the project. I will not look any further into the legislation of each law/act, as that would get very very messy.
There is a number of publications that link to this wide array of documents. This is the wider array of Documents relating to the JTTP 4-05:
This is the extent of what I can find with the sources I currently have available. This is a wide range of resources and more than enough to scale up the entire project to a much more comprehensive document navigator.