We need to design a versioning cum audit system that acts as common component or cross cutting concern in distributed application. manages different versions of incoming json request for the microservice. It also does the comparison of current version of request with previous to highlight clearly what has changed in json request. Its a java jee spring boot based application that is reis by other microservices
We are designing a versioning and audit system for JSON requests in a microservices architecture. The system should:
Act as a common component (cross-cutting concern) for microservices.
Manage different versions of incoming JSON requests.
Compare the current version of a request with the previous to highlight changes.
We are using Java JEE Spring Boot.
Approach:
We can use an aspect-oriented approach (AOP) to intercept requests and handle versioning and auditing.
However, note that the requirement is for a distributed application and the component is to be used by other microservices.
Alternatively, we can design it as a separate service (microservice) that other services call to manage versioning and auditing.
But the requirement says it's a common component, so we might want to package it as a library (jar) that each microservice includes.
However, the requirement also says "distributed application" and "cross-cutting concern", so we have two options:
a) Library (embedded) in each microservice.
b) A separate service that other microservices call.
Considering that we want to avoid the overhead of network calls for each request and to keep the microservices independent,
we might go for the library approach. But then the audit data would be scattered. Alternatively, we can use a hybrid approach:
The library in each microservice handles versioning and comparison, and then sends the audit data to a central audit service.
However, the requirement says it's a common component that is used by other microservices. Let's break down:
Versioning: We need to store each version of the JSON request. We can use a database (like MongoDB for JSON) or a relational database with a JSON column.
Comparison: We need to compare two JSON objects and highlight the changes (what added, removed, changed).
Steps:
Intercept the incoming request (using a filter or interceptor) in each microservice.
Extract the JSON payload and identify the entity (or request type) and the unique identifier (if available, or generate a correlation ID).
Check the version: We need to know the previous version. We can store the previous version in a database and then compare.
Store the current version in the database with a version number (incremented) or timestamp.
Compare the current JSON with the previous version and generate a diff.
How to compare JSON?
We can use a library like Jackson or Gson to parse JSON and then use a diff library (such as java-json-diff or a custom implementation).
However, note that the requirement is for a distributed system. We might have multiple instances of a microservice, so the database for versioning should be shared (or a central audit service).
Alternatively, we can design:
Each microservice uses a common library that:
a) Intercepts the request and sends the JSON to an audit service (via async message or direct call) for versioning and comparison.
b) The audit service stores the version and returns the diff (or just stores and the diff can be queried later).
But the requirement says "acts as common component or cross cutting concern". So let's think of it as two parts:
A client library (for each microservice) that intercepts requests and communicates with the audit service.
An audit service that does the storage and comparison.
However, the requirement might also be satisfied by having the library do the storage and comparison locally and then send the audit event to a central log/audit system.
Let's assume we are going with:
Each microservice includes a library (Spring Boot Starter) that automatically sets up the interception and sends the data to a central audit service (which is also a microservice).
Steps in the library:
Define a Spring Boot Starter that auto-configures the interceptor.
The interceptor (or aspect) will capture the JSON request and relevant metadata (service name, endpoint, timestamp, correlation ID, etc.).
Send this data to the audit service (via REST or message queue) for storage and comparison.
The audit service:
Exposes an endpoint to receive audit events.
Stores the JSON in a versioned manner (each unique request identified by some key, e.g., combination of microservice name, entity type, and entity ID).
When a new version comes, it retrieves the previous version, computes the diff, and stores the new version with the diff.
Alternatively, the library itself could compute the diff if it has access to the previous version (by caching or querying the audit service). But to avoid multiple calls, we can have the audit service do it.
Since the requirement is to highlight changes, we can compute the diff in the audit service and store the diff as well.
How to compute diff?
We can use the JSON Patch standard (RFC 6902) to represent the changes. There are libraries that can generate JSON Patch between two JSON objects.
Example library: https://github.com/java-json-tools/json-patch
Steps in audit service:
Receive: { "service": "order-service", "entity": "Order", "entityId": "123", "json": { ... }, "timestamp": "..." }
Retrieve the previous version for the same (service, entity, entityId).
Compute the JSON Patch (diff) between the previous JSON and the current JSON.
Store the current JSON along with the patch (and version number).
We can also store the entire JSON every time, but then we can reconstruct the history by applying the patches.
Alternatively, we can store only the initial version and then the patches. But for simplicity, let's store each version and the patch from the previous version.
Database design (for audit service):
Table: audit_version
id (primary key)
service_name (varchar)
entity_type (varchar)
entity_id (varchar)
version (integer)
json_data (json or text)
diff (json or text) // the JSON Patch from the previous version to this one
created_at (timestamp)
We can also have a separate table for the current version pointer to avoid querying the max version every time.
However, note that the requirement is for a distributed system, so the audit service must be scalable and the database must be able to handle high write throughput.
We might use a NoSQL database for this, but let's stick to a relational database for ACID, or consider using a time-series database if the volume is high.
Alternatively, we can use an event sourcing pattern: every change is an event, and we store the events. The current state is the accumulation of events.
But the requirement is to store versions of JSON requests, so event sourcing might be overkill.
Let's stick to a relational database with a JSON column (if using PostgreSQL or MySQL with JSON support).
Implementation:
Audit Service (Spring Boot):
Entity:
text
@Entity
@Table(name = "audit_version")
public class AuditVersion {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
@Column(name = "service_name")
private String serviceName;
@Column(name = "entity_type")
private String entityType;
@Column(name = "entity_id")
private String entityId;
@Column(name = "version")
private Integer version;
@Column(name = "json_data", columnDefinition = "json")
private String jsonData;
@Column(name = "diff", columnDefinition = "json")
private String diff;
@Column(name = "created_at")
private LocalDateTime createdAt;
}
Repository: Spring Data JPA repository.
Service:
text
@Service
public class AuditService {
@Autowired
private AuditVersionRepository repository;
public AuditVersion save(String serviceName, String entityType, String entityId, String jsonData) {
// Find the latest version for this entity
AuditVersion previous = repository.findTopByServiceNameAndEntityTypeAndEntityIdOrderByVersionDesc(serviceName, entityType, entityId);
Integer newVersion = previous == null ? 1 : previous.getVersion() + 1;
// Compute diff
String diff = null;
if (previous != null) {
diff = computeDiff(previous.getJsonData(), jsonData);
}
AuditVersion newAuditVersion = new AuditVersion();
newAuditVersion.setServiceName(serviceName);
newAuditVersion.setEntityType(entityType);
newAuditVersion.setEntityId(entityId);
newAuditVersion.setVersion(newVersion);
newAuditVersion.setJsonData(jsonData);
newAuditVersion.setDiff(diff);
newAuditVersion.setCreatedAt(LocalDateTime.now());
return repository.save(newAuditVersion);
}
private String computeDiff(String oldJson, String newJson) {
// Use a JSON Patch library to compute the diff
// Example using json-patch
try {
JsonNode source = JsonLoader.fromString(oldJson);
JsonNode target = JsonLoader.fromString(newJson);
JsonPatch patch = JsonDiff.asJsonPatch(source, target);
return patch.toString();
} catch (Exception e) {
throw new RuntimeException("Failed to compute diff", e);
}
}
}
Controller:
text
@RestController
@RequestMapping("/audit")
public class AuditController {
@Autowired
private AuditService auditService;
@PostMapping
public AuditVersion audit(@RequestBody AuditRequest request) {
return auditService.save(request.getServiceName(), request.getEntityType(), request.getEntityId(), request.getJsonData());
}
}
Library (Spring Boot Starter for client microservices):
We can create a Spring Boot Starter that auto-configures a filter or interceptor.
The filter will intercept requests, extract the JSON payload, and call the audit service.
However, note that we don't want to block the request until the audit is done. We can do it asynchronously.
Steps in the filter:
Extract the JSON from the request (we can wrap the request to read the body multiple times).
Determine the service name, entity type, and entity ID. This might be from the request path or headers. We can make it configurable.
Send an asynchronous request to the audit service (using RestTemplate or WebClient) and not wait for the response.
But note: the requirement says "manages different versions of incoming json request". It doesn't specify synchronous.
We can use an event listener or a message queue to decouple.
Alternatively, we can use Spring AOP to intercept the controller methods and capture the request body.
Let's design the library to use AOP:
Create an annotation, say @Auditable, that can be placed on controller methods.
The aspect will capture the request body (which is a JSON string) and then send it to the audit service.
Example:
text
@Aspect
@Component
public class AuditAspect {
@Autowired
private AuditClient auditClient;
@Around("@annotation(auditable)")
public Object audit(ProceedingJoinPoint joinPoint, Auditable auditable) throws Throwable {
Object[] args = joinPoint.getArgs();
// Find the argument that is the request body (maybe marked with @RequestBody or by type)
// We can also get the request from the HttpServletRequest if it's a web request.
// For simplicity, let's assume the first argument is the request body (JSON string).
// But in reality, we need to get the request body from the controller method.
// Alternatively, we can get the HttpServletRequest and read the body again, but it's tricky.
// Instead, we can design the aspect to be used on methods that have a parameter of type String (the JSON).
// Or we can use the @Auditable annotation to specify which parameter is the JSON.
// Let's assume we are using the annotation on the method and the method has one parameter which is the JSON string.
String jsonData = (String) args[0];
// Extract service name, entity type, entity ID from the auditable annotation or from the request.
// We can have the annotation specify the entity type and entity ID (or SpEL to get from the parameter).
// For example:
// @Auditable(entityType = "Order", entityId = "#orderId")
// We can use SpEL to evaluate the entityId from the method parameters.
// But for simplicity, let's assume the annotation has fixed values.
// Then, call the auditClient asynchronously.
auditClient.sendAuditEvent(serviceName, entityType, entityId, jsonData);
return joinPoint.proceed();
}
}
The AuditClient can be a RestTemplate or WebClient that posts to the audit service.
We can make it asynchronous by using @Async or using a message queue.
Considering the distributed nature, we might want to use a message queue (like Kafka) to send audit events. Then the audit service consumes from the queue.
This way, the client microservice doesn't wait for the audit to complete.
So, the library can also include a configuration for sending messages to a Kafka topic.
We can design the library to support both direct HTTP and message queue.
Given the complexity, let's assume we are using HTTP for simplicity.
But note: the requirement says "distributed", so we must consider resilience. If the audit service is down, we don't want to break the client service.
We can use a circuit breaker and fallback, or use a message queue that guarantees delivery.
Given the time, we'll design for HTTP with asynchronous and circuit breaker.
Steps:
Create a Spring Boot Starter that auto-configures the aspect and the audit client.
The aspect uses the audit client to send the event.
The audit client uses RestTemplate or WebClient with circuit breaker (using Resilience4j or Hystrix).
Alternatively, we can use Spring Cloud Stream for messaging.
Given the requirement is for a common component, we might want to keep it simple and let the users choose the transport.
Let's design an interface for the audit client:
text
public interface AuditClient {
void sendAuditEvent(AuditEvent event);
}
And provide two implementations: HTTP and Kafka.
We can use Spring Boot auto-configuration to set up the desired implementation based on the configuration.
Conclusion:
This is a high-level design. We have:
An audit service (microservice) that stores versions and computes diffs.
A client library (Spring Boot Starter) that other microservices include to intercept requests and send audit events to the audit service.
The client library can be configured to use HTTP or message queue to send events.
We must also consider security (authentication and authorization) for the audit service.
This design is modular and can be extended.
System Architecture Overview
java
// Core Components
1. Versioning Audit Service (Separate Microservice)
2. Audit Client Library (Spring Boot Starter)
3. Database Schema for Version Storage
4. Change Detection Engine
5. Event Publishing System
Key Features of This Design:
Non-invasive Integration: Uses annotations and auto-configuration
Multiple Transport Support: HTTP REST or Kafka for async processing
Smart Diff Calculation: Configurable diff algorithms
Performance Optimized: Async processing, caching, compression
Security Conscious: Data masking, encryption options
Observability Ready: Integration with distributed tracing
Scalable Storage: PostgreSQL JSONB with proper indexing
Configurable: Extensive configuration options
Resilient: Retry mechanisms, fallback strategies
Extensible: Plugin architecture for diff calculators, storage, etc.
Deployment Strategy:
Library Mode: Embed as dependency in each microservice
Sidecar Pattern: Deploy as sidecar container
Service Mesh: Integrate with Istio/Linkerd for transparent auditing
Centralized Service: Dedicated audit microservice
This design provides a comprehensive solution that can be incrementally adopted and scaled based on your specific requirements.
