Apache Avro for Data Serialization: Efficient Data Handling in Kafka - Related to creating, avro, serialization:, logs, efficient

Writing Logs to Files in Node.js with the LogLayer

Writing Logs to Files in [website] with LogLayer.

Writing logs to files is a crucial aspect of application monitoring and debugging in production environments. In this article, we'll explore how to effectively manage file-based logging in [website] applications using LogLayer and the Log File Rotation Transport, a powerful and flexible logging library.

LogLayer is a modern, TypeScript-first logging library that provides a clean and intuitive API for structured logging, and it acts as a layer on top of your favorite logging library like winston or pino .

LogLayer provides a robust solution for file-based logging with several advantages:

Automatic log file rotation based on size or time.

Batching capabilities for improved performance.

npm install loglayer @loglayer/transport-log-file-rotation serialize-error Enter fullscreen mode Exit fullscreen mode.

Here's a basic example of setting up file logging:

import { LogLayer } from " loglayer " ; import { LogFileRotationTransport } from " @loglayer/transport-log-file-rotation " ; import { serializeError } from " serialize-error " ; const logger = new LogLayer ({ errorSerializer : serializeError , transport : [ new LogFileRotationTransport ({ filename : " ./logs/[website] " }), ], }); // Start logging! logger . withMetadata ({ port : 3000 , }). info ( " Application started " ); logger . withError ( new Error ( " Something went wrong " )) . error ( " An error occurred " ); Enter fullscreen mode Exit fullscreen mode.

LogLayer provides several ways to enrich your logs with additional information:

Context is persistent data that applies to all subsequent log entries:

import { hostname } from " node:os " ; const logger = new LogLayer ({ transport : [ new LogFileRotationTransport ({ filename : " ./logs/[website] " }), ], }); // Add context that will be included in all subsequent logs const contextLogger = logger . withContext ({ hostname : hostname (), environment : process . env . NODE_ENV , version : process . env . npm_package_version }); contextLogger . info ( " Server starting " ); // Will include context contextLogger . error ( " Connection failed " ); // Will include same context Enter fullscreen mode Exit fullscreen mode.

Metadata is one-time data that only applies to the current log entry:

// Add metadata for a single log entry logger . withMetadata ({ requestId : " 123 " , duration : 45 , statusCode : 200 }). info ( " Request processed " ); // Different metadata for different log entries logger . withMetadata ({ userId : " user-456 " , action : " login " }). info ( " User logged in " ); // Combine context and metadata const userLogger = logger . withContext ({ service : " user-service " }); userLogger . withMetadata ({ duration : 123 , status : " success " }). info ( " Operation completed " ); Enter fullscreen mode Exit fullscreen mode.

You can also configure static data at the transport level:

const logger = new LogLayer ({ transport : [ new LogFileRotationTransport ({ filename : " ./logs/[website] " , staticData : { hostname : hostname (), environment : process . env . NODE_ENV , version : process . env . npm_package_version } }), ], }); Enter fullscreen mode Exit fullscreen mode.

{ "level" : "info" , "message" : "Request processed" , "timestamp" : "[website]" , "hostname" : "prod-server-1" , "environment" : "production" , "version" : "[website]" , "requestId" : "123" , "duration" : 45 , "statusCode" : 200 } Enter fullscreen mode Exit fullscreen mode.

For applications that need to rotate logs daily:

const logger = new LogLayer ({ transport : [ new LogFileRotationTransport ({ filename : " ./logs/[website] " , frequency : " daily " , dateFormat : " YMD " , compressOnRotate : true , maxLogs : " 7d " // Keep logs for 7 days }), ], }); Enter fullscreen mode Exit fullscreen mode.

For high-volume applications, rotating based on file size:

const logger = new LogLayer ({ transport : [ new LogFileRotationTransport ({ filename : " ./logs/[website] " , size : " 10M " , // Rotate when file reaches 10 MB maxLogs : 5 , // Keep last 5 log files compressOnRotate : true }), ], }); Enter fullscreen mode Exit fullscreen mode.

For high-throughput applications, enable batching to reduce disk I/O:

const logger = new LogLayer ({ transport : [ new LogFileRotationTransport ({ filename : " ./logs/[website] " , batch : { size : 1000 , // Write after 1000 logs are queued timeout : 5000 // Or after 5 seconds, whichever comes first } }), ], }); Enter fullscreen mode Exit fullscreen mode.

Use Rotation: Always configure log rotation to prevent disk space issues:

new LogFileRotationTransport ({ filename : " ./logs/[website] " , frequency : " daily " , maxLogs : " 30d " , compressOnRotate : true }) Enter fullscreen mode Exit fullscreen mode.

Enable Batching for High Volume: For applications with high log volume:

new LogFileRotationTransport ({ filename : " ./logs/[website] " , batch : { size : 1000 , timeout : 5000 } }) Enter fullscreen mode Exit fullscreen mode.

Separate Logs by Concern: Use different transports for different types of logs:

const logger = new LogLayer ({ transport : [ new LogFileRotationTransport ({ filename : " ./logs/[website] " }), new LogFileRotationTransport ({ filename : " ./logs/[website] " , levelMap : { error : " ERROR " , fatal : " FATAL " } }) ], }); Enter fullscreen mode Exit fullscreen mode.

// Use context for values that remain constant const serviceLogger = logger . withContext ({ service : " user-service " , version : " [website] " }); // Use metadata for request-specific information serviceLogger . withMetadata ({ requestId : " req-123 " , duration : 45 , statusCode : 200 }). info ( " Request completed " ); Enter fullscreen mode Exit fullscreen mode.

Proper error serialization is crucial for debugging:

import { serializeError } from " serialize-error " ; const logger = new LogLayer ({ errorSerializer : serializeError , transport : [ new LogFileRotationTransport ({ filename : " ./logs/[website] " }), ], }); try { throw new Error ( " Something went wrong " ); } catch ( error ) { logger . withError ( error ). error ( " Operation failed " ); } Enter fullscreen mode Exit fullscreen mode.

LogLayer provides a powerful and flexible solution for file-based logging in [website] applications. With aspects like automatic rotation, compression, and batching, it helps you maintain clean and efficient logs while ensuring you don't miss critical information for debugging and monitoring.

For more detailed information about LogLayer's file rotation transport, check out:

APIs have steadily become the backbone of AI systems, connecting data and tools seamlessly. Discover how they can drive scalable and secure training f......

Look Closer, Inspiration Lies Everywhere (February 2025 Wallpapers Edition).

Custom Hooks in React: Creating Reusable Logic with Examples

React’s custom hooks are a powerful feature that allows developers to encapsulate and reuse logic across components. By creating custom hooks, you can avoid code duplication, improve readability, and make your components more focused on rendering rather than logic. In this article, we’ll explore how to create custom hooks, with practical examples like useFetch , useLocalStorage , and useAuth . We’ll also discuss best practices and provide useful resources for further learning.

Custom hooks are JavaScript functions that start with the prefix use and can call other hooks ([website], useState , useEffect ). They allow you to extract component logic into reusable functions. Custom hooks follow the same rules as built-in hooks:

They must start with the word use . They can only be called at the top level of a functional component or another hook.

Reusability: Share logic across multiple components. Separation of Concerns: Keep components focused on rendering. Readability: Simplify complex components by moving logic into hooks. Testability: Isolate logic for easier testing.

A common use case is fetching data from an API. Instead of writing the same logic in multiple components, you can create a useFetch hook.

import { useState, useEffect } from 'react'; function useFetch(url) { const [data, setData] = useState(null); const [loading, setLoading] = useState(true); const [error, setError] = useState(null); useEffect(() => { const fetchData = async () => { try { const response = await fetch(url); if (![website] throw new Error('Network response was not ok'); const result = await [website]; setData(result); } catch (error) { setError(error); } finally { setLoading(false); } }; fetchData(); }, [url]); return { data, loading, error }; }.

function App() { const { data, loading, error } = useFetch('[website]'); if (loading) return Loading...; if (error) return Error: {error.message}; return ( {data?.map((post) => ( {[website]} ))} ); }.

Another common use case is persisting data in the browser’s local storage. The useLocalStorage hook simplifies this process.

import { useState } from 'react'; function useLocalStorage(key, initialValue) { const [storedValue, setStoredValue] = useState(() => { try { const item = window.localStorage.getItem(key); return item ? [website] : initialValue; } catch (error) { [website]; return initialValue; } }); const setValue = (value) => { try { const valueToStore = value instanceof Function ? value(storedValue) : value; setStoredValue(valueToStore); window.localStorage.setItem(key, JSON.stringify(valueToStore)); } catch (error) { [website]; } }; return [storedValue, setValue]; }.

function App() { const [name, setName] = useLocalStorage('name', 'John Doe'); return ( setName([website]} /> Hello, {name}! ); }.

Managing authentication state is a common requirement in applications. The useAuth hook can handle login, logout, and user state.

import { useState } from 'react'; function useAuth() { const [user, setUser] = useState(null); const login = (userData) => { setUser(userData); localStorage.setItem('user', JSON.stringify(userData)); }; const logout = () => { setUser(null); localStorage.removeItem('user'); }; return { user, login, logout }; }.

function App() { const { user, login, logout } = useAuth(); return ( {user ? ( Welcome, {[website]}! Logout ) : ( login({ name: 'John Doe' })}>Login )} ); }.

Custom hooks in React are a powerful way to encapsulate and reuse logic across components. They help keep your code DRY (Don’t Repeat Yourself) and make your components more focused and readable. However, to ensure your custom hooks are effective and maintainable, it’s essential to follow best practices. Below, we’ll outline these best practices in a table format for easy reference, along with necessary implementations.

Here’s a table summarizing the best practices for creating and using custom hooks:

Best Practice Description Example Prefix with use Always start the hook name with use to follow React’s naming conventions. useFetch , useLocalStorage , useAuth Single Responsibility Each hook should handle one specific piece of logic. useFetch for data fetching, useLocalStorage for local storage management. Document the Hook Add comments or documentation to explain the purpose and usage of the hook. Include a description of inputs, outputs, and side effects. Test Custom Hooks Test hooks in isolation using libraries like React Testing Library. Write unit tests for useFetch to ensure it handles loading and errors. Avoid Overuse Only create custom hooks when the logic is reusable across multiple components. Avoid creating hooks for one-off logic. Handle Edge Cases Ensure your hook handles edge cases like errors, null values, or missing data. In useFetch , handle network errors and invalid responses. Keep Hooks Small Break down complex logic into smaller, reusable hooks. Split useAuth into useLogin and useLogout if needed. Use Dependency Arrays Properly use dependency arrays in useEffect to avoid unnecessary re-renders. In useFetch , include url in the dependency array.

React Documentation: Custom Hooks useHooks: A collection of reusable custom hooks ([website] React Testing Library: Testing Custom Hooks Baeldung Tutorials: Creating Custom Hooks GitHub Repositories: Explore open-source custom hooks on GitHub for inspiration.

By leveraging custom hooks, you can write cleaner, more maintainable React code. Whether you’re fetching data, managing local storage, or handling authentication, custom hooks provide a flexible and reusable solution. Start creating your own hooks today and take your React development to the next level!

Jumpstarting your career as a community manager.

Managing communities has always been something I wanted to explore. This year, I finally decided to t......

The Case For Minimal WordPress Setups: A Contrarian View On Theme Frameworks.

Microsoft has not long ago launched Azure AI Foundry Labs, a hub designed to bring the latest AI research and experiments to developers, startups, and ent......

Apache Avro for Data Serialization: Efficient Data Handling in Kafka

In the world of data-driven applications, efficient data serialization is critical for performance, scalability, and interoperability. Apache Avro is a popular data serialization framework that excels in these areas, especially when used with Apache Kafka. Avro’s compact binary format, schema evolution capabilities, and seamless integration with Kafka make it a top choice for modern data pipelines. In this article, we’ll explore how to use Avro schemas for efficient data serialization in Kafka, compare Avro with Protocol Buffers (Protobuf) and JSON, and provide practical examples.

Apache Avro is a data serialization system that provides:

Compact Binary Format: Avro serializes data into a compact binary format, reducing storage and network overhead. Schema Evolution: Avro supports schema evolution, allowing you to modification schemas without breaking compatibility. Schema-Based Serialization: Avro uses schemas (defined in JSON) to serialize and deserialize data, ensuring type safety. Language Independence: Avro supports multiple programming languages, including Java, Python, and C++.

When working with Kafka, Avro offers several advantages:

Efficiency: Avro’s binary format is more compact than text-based formats like JSON, reducing Kafka’s storage and bandwidth requirements. Schema Management: Avro integrates with Schema Registry, a centralized repository for managing schemas and ensuring compatibility. Interoperability: Avro’s language-agnostic schemas enable seamless data exchange between systems written in different languages.

Avro schemas are defined in JSON. Here’s an example schema for a User record:

{ "type": "record", "name": "User", "fields": [ {"name": "id", "type": "int"}, {"name": "name", "type": "string"}, {"name": "email", "type": "string"} ] }.

Using the Avro schema, you can serialize data into a binary format. Here’s an example in Java:

import [website]; import [website]; import [website]; import [website]; import [website]; import [website]; import [website]; import [website]; public class AvroSerializer { public static byte[] serializeUser(Schema schema, int id, String name, String email) throws IOException { GenericRecord user = new [website]; [website]"id", id); [website]"name", name); [website]"email", email); ByteArrayOutputStream out = new ByteArrayOutputStream(); BinaryEncoder encoder = [website], null); SpecificDatumWriter writer = new SpecificDatumWriter<>(schema); [website], encoder); [website]; [website]; return out.toByteArray(); } }.

To deserialize the binary data back into a record:

import [website]; import [website]; import [website]; import [website]; import [website]; import [website]; import [website]; public class AvroDeserializer { public static GenericRecord deserializeUser(Schema schema, byte[] data) throws IOException { ByteArrayInputStream in = new ByteArrayInputStream(data); Decoder decoder = [website], null); SpecificDatumReader reader = new SpecificDatumReader<>(schema); return [website], decoder); } }.

Avro works seamlessly with Kafka when paired with a Schema Registry. The Schema Registry stores Avro schemas and ensures compatibility between producers and consumers.

Example: Producing Avro Messages to Kafka.

import [website]; import [website]; import [website]; import [website]; public class AvroKafkaProducer { public static void main(String[] args) { Properties props = new Properties(); [website]"bootstrap.servers", "localhost:9092"); [website]"key.serializer", "[website]"); [website]"value.serializer", [website]; [website]"[website]", "[website]:8081"); KafkaProducer producer = new KafkaProducer<>(props); GenericRecord user = new [website]; [website]"id", 1); [website]"name", "John Doe"); [website]"email", "[website]"); ProducerRecord record = new ProducerRecord<>("users", user); [website]; [website]; } }.

Example: Consuming Avro Messages from Kafka.

4. Comparing Avro with Protobuf and JSON.

Feature Avro Protobuf JSON Serialization Format Binary Binary Text-based Schema Evolution Excellent (supports full evolution) Good (requires careful design) None (no built-in schema) Performance High (compact, fast) High (compact, fast) Low (verbose, slower parsing) Human Readability No (binary format) No (binary format) Yes (text-based) Language Support Multiple (Java, Python, etc.) Multiple (Java, Python, etc.) Universal Schema Management Integrated with Schema Registry Requires external tools None.

Kafka Integration : Avro is ideal for Kafka due to its compact format and Schema Registry integration.

: Avro is ideal for Kafka due to its compact format and Schema Registry integration. Schema Evolution : Use Avro when you need to evolve schemas without breaking compatibility.

: Use Avro when you need to evolve schemas without breaking compatibility. High-Performance Systems: Avro’s binary format is perfect for systems requiring low latency and high throughput.

Apache Avro Documentation: [website] Confluent Schema Registry: [website] Protocol Buffers Documentation: [website] JSON Schema: [website]/ Kafka Avro Tutorial: [website].

By leveraging Apache Avro for data serialization in Kafka, you can achieve efficient, scalable, and interoperable data pipelines. Whether you’re building real-time streaming applications or batch processing systems, Avro’s compact format and schema evolution capabilities make it a powerful tool in your data engineering toolkit.

Read Part 1: 3 Key Benefits of Platform Engineering.

Having understood the principles and benefits of platform engineering, let’s take a deeper dive i......

It was never going to be long before Google got into the game of code assistance with Gemini. The headline is the number of completions being offered ......

Software security expert Tanya Janca, author of Alice and Bob Learn Secure Coding and Staff DevRel at AppSec business Semgrep, joins Ryan to talk about......