# bigdatatech

**Repository Path**: oscsc/bigdatatech

## Basic Information

- **Project Name**: bigdatatech
- **Description**: 大数据相关的测试验证
- **Primary Language**: Scala
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2023-03-07
- **Last Updated**: 2024-11-28

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# 大数据相关

## Calcite (TODO)

- [内存数据源和SQL解析](./calcite)

## Hadoop

[RPC](TODO)
- [Protobuf RPC](./hadoop/rpc/protobuf)
- [Writable RPC](./hadoop/rpc/writable)

## Spark 

[相关知识](https://xliuqq.github.io/blog_md/distributed_computing/spark)

- [Checkpoint和Cache](./spark/checkpoint)
- [集成SpringBoot](./spark/springboot)
- [自定义SQL V1 Source](./spark/sqlv1datasource)
- [Spark通过JNI在C++中加载Python并推理](./spark/sparkaiinfer/README.md)
- [Spark Metrics Customized Source](./spark/metrics-source/README.md)
- [Spark 串接流进度计算](./spark/module-progress/README.md)
- [Spark DataFrame的拖拽式机器学习后端](./spark/dataflowml/README.md)

## Flink

- [Checkpoint](./flink/checkpoint)

## Hudi
- [Spark示例](./hudi/spark)
- [Flink示例](./hudi/flink)

## [Fire Annotation](https://github.com/FireFramework/fire)

- [Spark 注解开发](./fireannotation/spark)

## 一致性

### Ratis
- [全局计数器](./consistency/ratis/src/main/java/com/xliu/cs/bigdatatech/consistency/ratis/counter)