Cloud computing

Introduction

Architecture de Spring Cloud

Eureka Config Serve Zuul Consul Hystrix Resilience4J

Spring Boot (BackEnd) TPs

Creation,Dépendance,Configuration Exemple Video RestController

Produit Rest API

Entity et repository Ajouter Afficher Liste Produit Détails,Supprimer,Vider Modifier

Angular (FrontEnd)

Angular Rappel CLient CRUD

Spring Security

User Auth

CRUD

Vente CRUD

To be Continued...

Middlewares Orientés Messages

Communication Synchrone vs. Asynchrone API JMS : Java Message Service JMS avec ActiveMQ et HornetQ KAFKA

Spring Batch

Spring Batch

Stream Processing

Kafka Streams

Architectures Serverless

Architectures Serverless Résumé



Batch processing

Batch processing is a method of running a series of tasks or processes on a set of data without manual intervention. It is designed to handle large volumes of data efficiently by grouping jobs into batches and processing them sequentially or in parallel.

Key Characteristics

  • Processes large volumes of data in batches

    Email Campaigns

    • Example: When sending a large number of emails (e.g., a marketing email campaign), email servers often use batch processing to send emails in groups. This prevents overloading the server and ensures emails are delivered efficiently.
    • Use Case: A company sends out promotional emails to thousands of customers. The emails are grouped into batches, and each batch is sent at specific intervals to avoid being flagged as spam.
  • scheduled to run at specific times (e.g., nightly jobs)

    Database Backups

    • Example: Many organizations schedule nightly backups of their databases to ensure data is safely stored without impacting user operations during peak hours.
    • Use Case: A company schedules a backup job every night at midnight to back up its customer database, including transaction logs, ensuring that it can restore the data in case of any failure.

    Automated Software Updates

    • Example: Systems and applications may schedule jobs to check for and install software updates at specific times to keep systems secure and up to date.
    • Use Case: A company schedules its IT system to run an update check every night at 3 AM, automatically downloading and installing critical security patches for its servers without disrupting users.
  • Reduces the need for constant manual monitoring

    Database Performance Optimization

    • Example: Databases can run maintenance jobs, like reindexing or defragmenting, at off-peak hours to maintain performance, reducing the need for manual monitoring or intervention.
    • Use Case: A database system is scheduled to automatically optimize performance every night by reorganizing tables and running integrity checks. This ensures that users experience optimal performance without the need for an administrator to manually intervene.

    Alert Management

    • Example: Automated systems can detect issues (e.g., system errors, performance drops, or security breaches) and generate alerts only when necessary, reducing the need for IT staff to monitor systems constantly.
    • Use Case: A monitoring system checks for website uptime every 5 minutes. If the site is down, it automatically sends an email or SMS alert to the support team. This eliminates the need for someone to manually check website status.

    Automated Fraud Detection

    • Example: Fraud detection systems can run periodic checks to analyze transactions for unusual patterns (e.g., high-value transfers or suspicious locations) without requiring constant human monitoring.
    • Use Case: A financial system runs a scheduled task every night to review transactions for potential fraud. If any suspicious activity is detected, an alert is sent to the fraud department for further investigation.
  • Optimized for high-throughput and low-latency requirements

    Real-Time Financial Transactions

    • Example: Payment processing systems are designed to handle thousands of transactions per second with minimal delay to ensure that customers’ payments are processed instantly.
    • Use Case: A credit card payment gateway is optimized for high throughput, processing thousands of transactions per second while ensuring that each payment authorization happens within milliseconds to avoid user frustration or potential loss of revenue.

Benefits of Batch Processing

  • Reduces manual intervention
  • Improves efficiency for repetitive tasks
  • Handles large datasets with ease
  • Cost-effective for long-running processes

Example of Jobs in Sales Management App

  1. Automated Alerts:

    • Job: Send an alert when stock is low.
    • Batch Process: Checks inventory levels at regular intervals and sends alerts via email or SMS when stock levels fall below a specified threshold.
  2. Stock Updates:

    • Job: Automatically update stock levels when orders are made.
    • Batch Process: After an order is processed, the stock quantity is automatically updated, reducing the inventory based on the items purchased.
  3. Order Processing:

    • Job: Process customer orders and update inventory.
    • Batch Process: Manages the entire process of receiving, validating, and processing customer orders, ensuring stock levels are updated accordingly.
  4. Daily Sales Report:

    • Job: Generate a daily sales report.
    • Batch Process: Collects data on all sales transactions for the day, aggregates totals, and produces a formatted sales report for management.
  5. Customer Invoicing:

    • Job: Generate invoices for completed orders.
    • Batch Process: Automatically generates invoices for customers after their orders are confirmed and processed.
  6. Price Updates:

    • Job: Update product prices based on certain criteria (e.g., cost price changes or promotional offers).
    • Batch Process: Periodically checks for price updates and applies changes to product listings.
  7. Inventory Replenishment:

    • Job: Automatically generate purchase orders when stock levels are low.
    • Batch Process: Analyzes stock levels and automatically generates purchase orders for products that need replenishment.
  8. Refund Processing:

    • Job: Process refunds for canceled or returned orders.
    • Batch Process: Handles the refund process by checking returned items, validating the transaction, and issuing refunds to customers.
  9. Customer Feedback Collection:

    • Job: Collect customer feedback and generate reports.
    • Batch Process: Periodically pulls in customer feedback forms, aggregates responses, and creates reports for analysis.
  10. Expired Product Removal:

    • Job: Remove expired products from the inventory system.
    • Batch Process: Scans the inventory for products with expiration dates and automatically removes them from the stock list.
  11. Discount Application:

    • Job: Apply discounts for specific products or customer groups.
    • Batch Process: Periodically applies discounts or special pricing to eligible products based on promotional rules.
  12. Email Newsletter Sending:

    • Job: Send promotional emails or newsletters to customers.
    • Batch Process: Automatically sends marketing emails to customers on a regular schedule with promotional offers, new arrivals, etc.
  13. Sales Tax Calculation:

    • Job: Calculate sales tax for orders based on customer location.
    • Batch Process: Calculates and applies the correct sales tax rate to customer orders.
  14. Customer Account Inactivity Alerts:

    • Job: Notify customers with inactive accounts.
    • Batch Process: Periodically checks for inactive customer accounts and sends alerts encouraging them to make a new purchase.
  15. Order Shipment Processing:

    • Job: Process shipments for completed orders.
    • Batch Process: Automatically updates the status of orders that are ready for shipment, generating shipping labels and tracking information.

Batch keywords

  • Job : Defines the overall batch process.
  • Step: Represents a unit of work within a job.
  • ItemReader: Reads data from a source (e.g., a database).
  • ItemProcessor: Processes the data (e.g., transforms or filters).
  • ItemWriter: Writes processed data to a destination (e.g., a database, file, or email).
  • Chunk: Groups items together for efficient processing in bulk.
  • JobLauncher: Launches and manages the execution of jobs.
  • @Scheduled: Used for scheduling jobs to run periodically.
graph TD A[Job] --> B[Step] B --> C[ItemReader] C --> D[ItemProcessor] D --> E[ItemWriter] B --> F[Chunk] F --> G[JobExecution] subgraph Batch_Process A B C D E F end %% Job Launcher and Execution H[JobLauncher] -->|Launches| A A --> I[JobExecution Status] I --> J[Success / Failure] %% Optional Scheduling K[Scheduled Task] -->|Triggers Job| A %% External System Trigger L[External System] -->|Triggers Job| K

Sales Management Jobs

graph TD A[Job: automatedAlertJob] --> B[Step: automatedAlertStep] B --> C[ItemReader: stockLowProductReader] C --> D[ItemProcessor: stockLowProductProcessor] D --> E[ItemWriter: stockLowProductWriter] B --> F[Chunk 10] F --> G[JobExecution] subgraph Batch_Process A B C D E F end %% Scheduling Job Execution H[Scheduled Task] -->|Triggers Job| A %% Optional components G --> I[JobLauncher] I --> A %% Status Monitoring J[JobExecution Status] --> G J --> K[Success / Failure] %% External triggers H --> L[External System] --> H

1. Job (Job)

Definition: A Job represents a batch process in Spring Batch. It's the highest level of abstraction that encapsulates a series of steps. A job can be thought of as a unit of work that executes multiple steps in a specific sequence.
Purpose: It is the container for the entire batch process.
Example:

@Bean
public Job automatedAlertJob() {
    return jobBuilderFactory.get("automatedAlertJob")
            .start(automatedAlertStep())
            .build();
}
Explanation: This job starts by executing the automatedAlertStep and then completes. A Job can be complex and involve multiple steps, which will be executed sequentially or conditionally.

2. Step (Step)

Definition: A Step in Spring Batch is a single phase of a batch job that is executed sequentially. A step can perform a variety of tasks, such as reading data, processing it, and writing the results.
Purpose: It is a smaller unit within a Job that executes a specific task like reading data, transforming it, or saving it.
Example:

@Bean
public Step automatedAlertStep() {
    return stepBuilderFactory.get("automatedAlertStep")
            .<Product, Product>chunk(10)
            .reader(stockLowProductReader())
            .processor(stockLowProductProcessor())
            .writer(stockLowProductWriter())
            .build();
}
Explanation: Here, a step named automatedAlertStep reads Product entities, processes them (checks if their stock is low), and writes an alert. The chunk(10) defines that the step processes items in chunks of 10.

3. ItemReader (ItemReader<T>)

Definition: ItemReader is an interface used to define the reading behavior in Spring Batch. It is responsible for reading input data, one item at a time.
Purpose: It reads data from a source (e.g., a database, file, or queue).
Example:

@Bean
public ItemReader<Product> stockLowProductReader() {
    return new ItemReader<Product>() {
        @Override
        public Product read() throws Exception {
            List<Product> products = productRepository.findByStockLessThan(10); // Threshold of 10 units
            return products.isEmpty() ? null : products.get(0);
        }
    };
}
Explanation: The stockLowProductReader reads a Product if its stock is less than 10. If no such product is found, it returns null, signaling the end of reading. The ItemReader abstracts the source of data from where records are read.

4. ItemProcessor (ItemProcessor<I, O>)

Definition: The ItemProcessor is responsible for processing the data read by the ItemReader. It takes an item as input, processes it (e.g., transforms, validates), and outputs a new item.
Purpose: It is used to transform or filter the data.
Example:

@Bean
public ItemProcessor<Product, Product> stockLowProductProcessor() {
    return product -> product; // Here, no processing, just pass the product unchanged
}
Explanation: In this case, the ItemProcessor doesn't modify the Product (it's simply passed unchanged), but typically it could be used to transform data, like calculating discounts, converting data formats, or filtering based on certain conditions.

5. ItemWriter (ItemWriter<T>)

Definition: ItemWriter is responsible for writing the processed items (usually after they're transformed or validated) to a destination like a database, file, or message queue.
Purpose: It writes processed data to the desired destination.
Example:

@Bean
public ItemWriter<Product> stockLowProductWriter() {
    return products -> {
        for (Product product : products) {
            sendAlertEmail(product);
        }
    };
}
Explanation: The stockLowProductWriter writes the data by sending an email alert for each Product whose stock is low. The ItemWriter handles the writing part, which could also involve saving data to a database or output file.

6. Chunk (chunk(int size))

Definition: A chunk is the basic concept of grouping a set of items to be processed together. Instead of processing one item at a time, Spring Batch processes multiple items (a chunk) in one transaction, improving performance.
Purpose: It allows processing of multiple items in one go to minimize I/O operations, such as database transactions or file writes.
Example:

@Bean
public Step automatedAlertStep() {
    return stepBuilderFactory.get("automatedAlertStep")
            .<Product, Product>chunk(10)  // This means that 10 items will be read, processed, and written together in a chunk
            .reader(stockLowProductReader())
            .processor(stockLowProductProcessor())
            .writer(stockLowProductWriter())
            .build();
}
Explanation: The chunk size 10 defines that 10 Product items are read, processed, and written in a single transaction. The chunk size can be adjusted based on system capacity and performance needs.

7. JobLauncher (JobLauncher)

Definition: The JobLauncher is used to launch a Job in Spring Batch. It allows jobs to be run programmatically or scheduled.
Purpose: It is responsible for executing the Job and managing the execution lifecycle.
Example:

@Autowired
private JobLauncher jobLauncher;

@Autowired
private Job automatedAlertJob;

@Scheduled(cron = "0 0 0 * * ?")
public void runAutomatedAlertJob() throws JobExecutionException {
    jobLauncher.run(automatedAlertJob, new JobParameters());
}
Explanation: The jobLauncher.run() method triggers the execution of a job. In this example, it's scheduled to run the automatedAlertJob every day at midnight.

8. JobExecution (JobExecution)

Definition: JobExecution represents the execution details of a Job. It provides information about the status of the job, such as whether it was successful or failed, and logs the execution data.
Purpose: It helps to monitor and manage job status, including its execution history.
Example: JobExecution is often used to check the status of the job or handle errors, but it's not explicitly shown in the code above.

9. @Scheduled (@Scheduled)

Definition: The @Scheduled annotation is used to schedule methods in Spring Boot, enabling batch jobs to run at fixed intervals or cron expressions.
Purpose: It automates the execution of tasks like batch jobs.
Example:

@Scheduled(cron = "0 0 0 * * ?") // Every day at midnight
public void runAutomatedAlertJob() throws JobExecutionException {
    jobLauncher.run(automatedAlertJob, new JobParameters());
}
Explanation: The @Scheduled annotation allows the batch job to run periodically. Here, it triggers the job every day at midnight.

Spring Boot Project with Spring Batch

dependencies in pom.xml.

<dependencies>
    <!-- Spring Boot and Batch Dependencies -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-batch</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-mail</artifactId> <!-- For sending emails -->
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
</dependencies>

2. Create Batch Jobs and Steps

Automated Alerts (Send an alert when stock is low)

This job sends an email alert if a product's stock is below a threshold.
AutomatedAlertsJobConfig.java

@Configuration
@EnableBatchProcessing
public class AutomatedAlertsJobConfig {

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    private ProductRepository productRepository; // Assuming Product is an entity

    @Autowired
    private JavaMailSender mailSender;

    @Bean
    public Job automatedAlertJob() {
        return jobBuilderFactory.get("automatedAlertJob")
                .start(automatedAlertStep())
                .build();
    }

    @Bean
    public Step automatedAlertStep() {
        return stepBuilderFactory.get("automatedAlertStep")
                .<Product, Product>chunk(10)
                .reader(stockLowProductReader())
                .processor(stockLowProductProcessor())
                .writer(stockLowProductWriter())
                .build();
    }

    @Bean
    public ItemReader<Product> stockLowProductReader() {
        return new ItemReader<Product>() {
            @Override
            public Product read() throws Exception {
                List<Product> products = productRepository.findByStockLessThan(10); // Threshold of 10 units
                return products.isEmpty() ? null : products.get(0);
            }
        };
    }

    @Bean
    public ItemProcessor<Product, Product> stockLowProductProcessor() {
        return product -> product; // You can add any logic here if needed
    }

    @Bean
    public ItemWriter<Product> stockLowProductWriter() {
        return products -> {
            for (Product product : products) {
                sendAlertEmail(product);
            }
        };
    }

    private void sendAlertEmail(Product product) {
        SimpleMailMessage message = new SimpleMailMessage();
        message.setTo("admin@company.com");
        message.setSubject("Low Stock Alert: " + product.getName());
        message.setText("The stock for product " + product.getName() + " is low. Please restock.");
        mailSender.send(message);
    }
}

Stock Updates (Automatically update stock when orders are made)

batch job that updates stock when orders are placed.
StockUpdateJobConfig.java

@Configuration
@EnableBatchProcessing
public class StockUpdateJobConfig {

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    private OrderRepository orderRepository; // Assuming Order is an entity

    @Autowired
    private ProductRepository productRepository;

    @Bean
    public Job stockUpdateJob() {
        return jobBuilderFactory.get("stockUpdateJob")
                .start(stockUpdateStep())
                .build();
    }

    @Bean
    public Step stockUpdateStep() {
        return stepBuilderFactory.get("stockUpdateStep")
                .<Order, Product>chunk(10)
                .reader(orderReader())
                .processor(orderProcessor())
                .writer(stockUpdateWriter())
                .build();
    }

    @Bean
    public ItemReader<Order> orderReader() {
        return new ItemReader<Order>() {
            @Override
            public Order read() throws Exception {
                List<Order> orders = orderRepository.findUnprocessedOrders();
                return orders.isEmpty() ? null : orders.get(0);
            }
        };
    }

    @Bean
    public ItemProcessor<Order, Product> orderProcessor() {
        return order -> {
            Product product = productRepository.findById(order.getProductId());
            product.setStock(product.getStock() - order.getQuantity());
            return product;
        };
    }

    @Bean
    public ItemWriter<Product> stockUpdateWriter() {
        return products -> productRepository.saveAll(products);
    }
}

Daily Sales Report Generation

This job generates a daily report.
SalesReportJobConfig.java

@Configuration
@EnableBatchProcessing
public class SalesReportJobConfig {

    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    private SalesRepository salesRepository;

    @Bean
    public Job salesReportJob() {
        return jobBuilderFactory.get("salesReportJob")
                .start(salesReportStep())
                .build();
    }

    @Bean
    public Step salesReportStep() {
        return stepBuilderFactory.get("salesReportStep")
                .<Sale, String>chunk(10)
                .reader(salesReportReader())
                .processor(salesReportProcessor())
                .writer(salesReportWriter())
                .build();
    }

    @Bean
    public ItemReader<Sale> salesReportReader() {
        return salesRepository::findAll; // Assuming `Sale` is an entity
    }

    @Bean
    public ItemProcessor<Sale, String> salesReportProcessor() {
        return sale -> "Sale ID: " + sale.getId() + ", Amount: " + sale.getAmount(); // Customize report format
    }

    @Bean
    public ItemWriter<String> salesReportWriter() {
        return report -> {
            report.forEach(System.out::println); // You can replace with saving to a file or sending via email
        };
    }
}

Main Application Entry

MainApplication.java

@SpringBootApplication
public class SalesManagementApplication {

    public static void main(String[] args) {
        SpringApplication.run(SalesManagementApplication.class, args);
    }

}

Database Model


@Entity
public class Product {
    @Id
    private Long id;
    private String name;
    private int stock;
    // getters and setters
}

@Entity
public class Order {
    @Id
    private Long id;
    private Long productId;
    private int quantity;
    // getters and setters
}

@Entity
public class Sale {
    @Id
    private Long id;
    private double amount;
    // getters and setters
}

Cron Scheduling (Optional)

To schedule jobs to run at specific intervals (e.g., every day at midnight), you can use @Scheduled annotation:

@Configuration
@EnableScheduling
public class BatchSchedulingConfig {

    @Autowired
    private JobLauncher jobLauncher;

    @Autowired
    private Job automatedAlertJob;

    @Scheduled(cron = "0 0 0 * * ?") // Every day at midnight
    public void runAutomatedAlertJob() throws JobExecutionException {
        jobLauncher.run(automatedAlertJob, new JobParameters());
    }
}
  • Job: Defines the overall batch process.
  • Step: Represents a unit of work within a job.
  • ItemReader: Reads data from a source (e.g., a database).
  • ItemProcessor: Processes the data (e.g., transforms or filters).
  • ItemWriter: Writes processed data to a destination (e.g., a database, file, or email).
  • Chunk: Groups items together for efficient processing in bulk.
  • JobLauncher: Launches and manages the execution of jobs.
  • @Scheduled: Used for scheduling jobs to run periodically.





graph TD A[Spring Batch Configuration] --> B[Batch Processing] B --> C[Job Management] C --> D[Steps] D --> E[Job Repository] E --> F[Error Handling] F --> G[Partitioning] G --> H[Task Scheduling] H --> I[Integration with Data Sources] I --> J[Batch Job Testing] J --> K[Use of Listeners] A -->|Configures Spring Batch setup| B B -->|Manages and executes batch jobs| C C -->|Defines steps to be executed| D D -->|Stores information about job status and executions| E E -->|Handles errors in job execution| F F -->|Splits data processing across multiple partitions| G G -->|Schedules batch jobs to run at specific intervals| H H -->|Integrates with data sources e.g., databases| I I -->|Tests and verifies batch jobs execution| J J -->|Listens for job events and executes custom logic| K