Custom Data Source

In the Spring Batch ecosystem, core components such as ItemReader, ItemWriter, LineAggregator, and ChunkListener collectively form the backbone of batch processing logic. However, these interfaces often provide default implementations or relatively generic abstraction layers. When business scenarios involve complex database structures, special storage formats (such as JSON/CSV), or non-relational data sources, standard implementations may fail to meet requirements. In such cases, deeply understanding and mastering how to write custom ItemReader and ItemWriter becomes a critical step for technical advancement.
The design philosophy of Spring Batch lies in decoupling data processing logic from specific storage details. Through dependency injection, the framework delegates data reading, transformation, and writing tasks to Beans that implement specific interfaces. Default data sources typically rely on JDBC operations and are suitable for standard SQL relational database scenarios. However, in practical applications, we may encounter situations such as:

  • Non-standard Data Structures: Handling nested JSON objects or specific XML formatted data.
  • Special Performance Requirements: Pre-aggregating, filtering, or performing complex calculations in memory before reading/writing data.
  • Heterogeneous Multi-source Integration: Simultaneously reading from a mix of file systems and databases, or writing results to NoSQL document databases.

In these scenarios, directly operating on Spring Batch's abstract interfaces appears too coarse-grained and fails to reflect the uniqueness of business logic. Therefore, by inheriting ItemReader<T> or implementing its methods (as recommended for Java 8+), we can fully control the data flow processing process and encapsulate complex business rules within custom components.

Building a High-Performance ItemReader Example

Creating custom readers is a common requirement for handling complex source data. Below is an example implementation of JsonItemReader based on Spring Boot and the Jackson JSON library:

import org.springframework.batch.item.Item;
import org.springframework.batch.item.ItemReader;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

public class JsonItemReader implements ItemReader<String> {
    private final ObjectMapper objectMapper = new ObjectMapper();
    // Assuming an input source containing a list of JSON strings, demonstrated with mock data here
    private List<String> jsonStrings; 

    public void setJsonList(List<String> strings) {
        this.jsonStrings = strings;
    }

    @Override
    public Item read() throws IOException, InterruptedException {
        if (jsonStrings == null || jsonStrings.isEmpty()) {
            return null; // Return null to indicate end of reading
        }
        
        String jsonString = jsonStrings.remove(0);
        try {
            // Convert JSON string to Java Object, then wrap it into an Item
            com.fasterxml.jackson.databind.JsonNode node = objectMapper.readValue(jsonString, 
                com.fasterxml.jackson.databind.node.ObjectNode.class);
            
            return new JsonItem(node.toString());
        } catch (Exception e) {
            throw new RuntimeException("JSON Parsing Error", e);
        }
    }

    // Internal helper class example to encapsulate JSON data as an Item
    static class JsonItem implements Item {
        private String content;
        
        public JsonItem(String s) { this.content = s; }
        @Override public Object getPayload() { return content; }
        @Override public Class<?> getType() { return String.class; }
    }

    // Other Spring Batch interface methods (such as hasNext) can be implemented as needed
}

In this example, we do not directly manipulate database connection pools or JDBC drivers. Instead, we abstract the data source as a list of JSON strings. Through the definition of the read() method, we can easily perform preprocessing, deduplication, or skip certain records based on specific conditions during the reading process. This flexibility allows us to receive cleaned and transformed data objects in subsequent batch processing steps (Steps), greatly simplifying the complexity of downstream logic.

Implementing Flexible ItemWriter Strategies

Similar to readers, write operations also require highly customizable capabilities. Standard ItemWriters typically only handle adding single elements to a database collection. However, in actual business scenarios, we might need bulk inserts, transaction state recording, or execution of complex validation rules. Below is a CustomItemWriter implementing custom logic:

import org.springframework.batch.item.Item;
import org.springframework.batch.item.ItemWriter;
import java.util.List;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;

public class CustomItemWriter implements ItemWriter<String> {
    private Connection connection; // Inject database connection in constructor
    
    @Override
    public void write(List<? extends Item> items) throws Exception {
        if (items == null || items.isEmpty()) {
            return;
        }

        String sql = "INSERT INTO users (name, email) VALUES (?, ?);";
        
        try (PreparedStatement pstmt = connection.prepareStatement(sql)) {
            for (Item item : items) {
                Object payload = ((org.springframework.batch.item.support.SimplePayloadable<String>) item).getPayload();
                
                // Simulate business logic: check data validity or perform additional validation
                if (!isValidData(payload.toString())) {
                    throw new RuntimeException("Invalid data detected: " + payload);
                }

                pstmt.setString(1, (String) ((org.springframework.batch.item.support.SimplePayloadable<String>) item).getPayload());
                // Assuming the Item contains an email, adjustment is needed based on actual encapsulation
                
                pstmt.executeUpdate();
            }
        } catch (SQLException e) {
            throw new RuntimeException("Database Write Error", e);
        } finally {
            connection.close();
        }
    }

    private boolean isValidData(String data) {
        return data != null && !data.isEmpty() && data.length() > 0;
    }
}

It is worth noting that in real Spring Batch implementations, it is generally not recommended to manually manage the lifecycle of Connection objects. A more recommended approach is to inject the database connection into the custom Writer class (via dependency injection) or use AOP to intercept transaction boundaries. The code above demonstrates the core logic structure: receiving batch data, iterating to execute business validation, and finally performing data persistence operations.

Best Practices and Precautions

When developing custom components, adhering to the following principles can ensure the stability and maintainability of batch processing tasks:

  1. Avoid Resource Leaks: Regardless of how database connections or file streams are obtained, ensure they are properly closed after execution in read() or write() methods. Spring Batch's exception propagation mechanism may interrupt the entire job; therefore, all potential IO and SQL exceptions must be handled correctly.
  2. Thread Safety Considerations: Default Spring Batch tasks typically run in a multi-threaded environment (especially when using ParallelChunkProcessor). Custom components must be thread-safe; avoid maintaining shared state or mutable objects within instances. If business logic heavily relies on ordered processing, limit parallelism through configuration or use the singleton pattern with external locking mechanisms.
  3. Logging Standards: For large-scale data processing processes, appropriate logging output aids in troubleshooting. It is recommended to record reading progress (such as lines read) in the read() method, log detailed stack traces upon encountering errors, but avoid printing sensitive data.

By deeply mastering the techniques for writing custom ItemReader and ItemWriter, developers can break through the limitations of the Spring Batch framework and transform it into a powerful tool adaptable to various complex business scenarios. Whether processing massive unstructured data or building high-concurrency data processing pipelines, these core components serve as indispensable technical foundations.