add all code

rockXu · Dec 25, 2015 · 37cea6f · 37cea6f
1 parent 001b8a8
commit 37cea6f
Show file tree

Hide file tree

Showing 576 changed files with 58,011 additions and 1 deletion.
diff --git a/README b/README
diff --git a/README.md b/README.md
@@ -1,3 +1,101 @@
 # DataX
+
 DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。
-代码近期会上传，敬请期待。
+
+
+
+# Features
+
+DataX本身作为数据同步框架，将不同数据源的同步抽象为从源头数据源读取数据的Reader插件，以及向目标端写入数据的Writer插件，理论上DataX框架可以支持任意数据源类型的数据同步工作。同时DataX插件体系作为一套生态系统, 每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。
+
+
+
+# DataX详细介绍
+
+请参考：
+
+# System Requirements
+
+- Linux
+- [JDK(1.6以上，推荐1.6) ](http://www.oracle.com/technetwork/cn/java/javase/downloads/index.html) 
+- [Python(推荐Python2.6.X) ](https://www.python.org/downloads/)
+- [Apache Maven 3.x](https://maven.apache.org/download.cgi) (Compile DataX)
+
+# Quick Start
+
+请点击：[Quick Start](https://github.com/alibaba/DataX/wiki/Quick-Start)
+
+# Support Data Channels
+
+目前DataX支持的数据源有:
+
+### Reader
+
+> **Reader实现了从数据存储系统批量抽取数据，并转换为DataX标准数据交换协议，DataX任意Reader能与DataX任意Writer实现无缝对接，达到任意异构数据互通之目的。**
+
+**RDBMS 关系型数据库**
+
+- [MysqlReader](https://github.com/alibaba/DataX/blob/master/mysqlreader/doc/mysqlreader.md): 使用JDBC批量抽取Mysql数据集。
+- [OracleReader](https://github.com/alibaba/DataX/blob/master/oraclereader/doc/oraclereader.md): 使用JDBC批量抽取Oracle数据集。
+- [SqlServerReader](https://github.com/alibaba/DataX/blob/master/sqlserverreader/doc/sqlserverreader.md): 使用JDBC批量抽取SqlServer数据集
+- [PostgresqlReader](https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md): 使用JDBC批量抽取PostgreSQL数据集
+- [DrdsReader](https://github.com/alibaba/DataX/blob/master/drdsreader/doc/drdsreader.md): 针对公有云上DRDS的批量数据抽取工具。
+
+**数仓数据存储**
+
+- [ODPSReader](https://github.com/alibaba/DataX/blob/master/odpsreader/doc/odpsreader.md): 使用ODPS Tunnel SDK批量抽取ODPS数据。
+
+**NoSQL数据存储**
+
+- [OTSReader](https://github.com/alibaba/DataX/blob/master/otsreader/doc/otsreader.md): 针对公有云上OTS的批量数据抽取工具。
+- [HBaseReader](https://github.com/alibaba/DataX/blob/master/hbasereader/doc/hbasereader.md): 针对 HBase 0.94版本的在线数据抽取工具
+- [MongoDBReader](https://github.com/alibaba/DataX/blob/master/mongodbreader/doc/mongodbreader.md)：MongoDBReader
+
+**无结构化数据存储**
+
+- [TxtFileReader](https://github.com/alibaba/DataX/blob/master/txtfilereader/doc/txtfilereader.md): 读取(递归/过滤)本地文件。
+- [FtpReader](https://github.com/alibaba/DataX/blob/master/ftpreader/doc/ftpreader.md): 读取(递归/过滤)远程ftp文件。
+- [HdfsReader](https://github.com/alibaba/DataX/blob/master/hdfsreader/doc/hdfsreader.md): 针对Hdfs文件系统中textfile和orcfile文件批量数据抽取工具。 
+- [OssReader](https://github.com/alibaba/DataX/blob/master/ossreader/doc/ossreader.md): 针对公有云OSS产品的批量数据抽取工具。
+- StreamReader
+
+### Writer
+
+------
+
+> **Writer实现了从DataX标准数据交换协议，翻译为具体的数据存储类型并写入目的数据存储。DataX任意Writer能与DataX任意Reader实现无缝对接，达到任意异构数据互通之目的。**
+
+------
+
+**RDBMS 关系型数据库**
+
+- [MysqlWriter](https://github.com/alibaba/DataX/blob/master/mysqlwriter/doc/mysqlwriter.md): 使用JDBC(Insert,Replace方式)写入Mysql数据库
+- [OracleWriter](https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md): 使用JDBC(Insert方式)写入Oracle数据库
+- [PostgresqlWriter](https://github.com/alibaba/DataX/blob/master/postgresqlwriter/doc/postgresqlwriter.md): 使用JDBC(Insert方式)写入PostgreSQL数据库
+- [SqlServerWriter](https://github.com/alibaba/DataX/blob/master/sqlserverwriter/doc/sqlserverwriter.md): 使用JDBC(Insert方式)写入sqlserver数据库
+- [DrdsWriter](https://github.com/alibaba/DataX/blob/master/drdswriter/doc/drdswriter.md): 使用JDBC(Replace方式)写入Drds数据库
+
+**数仓数据存储**
+
+- [ODPSWriter](https://github.com/alibaba/DataX/blob/master/odpswriter/doc/odpswriter.md): 使用ODPS Tunnel SDK向ODPS写入数据。
+- [ADSWriter](https://github.com/alibaba/DataX/blob/master/adswriter/doc/adswriter.md): 使用ODPS中转将数据导入ADS。
+
+**NoSQL数据存储**
+
+- [OTSWriter](https://github.com/alibaba/DataX/blob/master/otswriter/doc/otswriter.md): 使用OTS SDK向OTS Public模型的表中导入数据。
+- [OCSWriter](https://github.com/alibaba/DataX/blob/master/ocswriter/doc/ocswriter.md)
+- [MongoDBWriter](https://github.com/alibaba/DataX/blob/master/mongodbwriter/doc/mongodbwriter.md)：MongoDBWriter
+
+**无结构化数据存储**
+
+- [TxtFileWriter](https://github.com/alibaba/DataX/blob/master/txtfilewriter/doc/txtfilewriter.md): 提供写入本地文件功能。
+- [OssWriter](https://github.com/alibaba/DataX/blob/master/osswriter/doc/osswriter.md): 使用OSS SDK写入OSS数据。
+- [HdfsWriter](https://github.com/alibaba/DataX/blob/master/hdfswriter/doc/hdfswriter.md): 提供向Hdfs文件系统中写入textfile文件和orcfile文件功能。
+- StreamWriter
+
+
+
+# Contact us
+
+请及时提出issue给我们。请前往：[DataxIssue](https://github.com/alibaba/DataX/issues)
+
diff --git a/adswriter/adswriter.iml b/adswriter/adswriter.iml
@@ -0,0 +1,72 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module org.jetbrains.idea.maven.project.MavenProjectsManager.isMavenModule="true" type="JAVA_MODULE" version="4">
+  <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_6" inherit-compiler-output="false">
+    <output url="file://$MODULE_DIR$/target/classes" />
+    <output-test url="file://$MODULE_DIR$/target/test-classes" />
+    <content url="file://$MODULE_DIR$">
+      <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
+      <sourceFolder url="file://$MODULE_DIR$/src/main/resources" type="java-resource" />
+      <excludeFolder url="file://$MODULE_DIR$/target" />
+    </content>
+    <orderEntry type="inheritedJdk" />
+    <orderEntry type="sourceFolder" forTests="false" />
+    <orderEntry type="module" module-name="datax-common" />
+    <orderEntry type="library" name="Maven: org.apache.commons:commons-lang3:3.3.2" level="project" />
+    <orderEntry type="library" name="Maven: com.google.guava:guava:18.0" level="project" />
+    <orderEntry type="library" name="Maven: com.alibaba:fastjson:1.1.43" level="project" />
+    <orderEntry type="library" name="Maven: commons-io:commons-io:2.4" level="project" />
+    <orderEntry type="library" name="Maven: org.apache.commons:commons-math3:3.1.1" level="project" />
+    <orderEntry type="module" module-name="datax-core" />
+    <orderEntry type="library" name="Maven: commons-cli:commons-cli:1.2" level="project" />
+    <orderEntry type="library" name="Maven: commons-beanutils:commons-beanutils:1.9.2" level="project" />
+    <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpclient:4.4" level="project" />
+    <orderEntry type="library" name="Maven: org.apache.httpcomponents:httpcore:4.4" level="project" />
+    <orderEntry type="library" name="Maven: commons-codec:commons-codec:1.9" level="project" />
+    <orderEntry type="library" name="Maven: org.apache.httpcomponents:fluent-hc:4.4" level="project" />
+    <orderEntry type="library" name="Maven: org.codehaus.janino:janino:2.5.16" level="project" />
+    <orderEntry type="module" module-name="plugin-rdbms-util" />
+    <orderEntry type="library" name="Maven: commons-collections:commons-collections:3.0" level="project" />
+    <orderEntry type="library" name="Maven: com.alibaba:druid:1.0.13" level="project" />
+    <orderEntry type="module-library">
+      <library name="Maven: com.alibaba:jconsole:1.8.0">
+        <CLASSES>
+          <root url="jar:///Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/lib/jconsole.jar!/" />
+        </CLASSES>
+        <JAVADOC />
+        <SOURCES />
+      </library>
+    </orderEntry>
+    <orderEntry type="module-library">
+      <library name="Maven: com.alibaba:tools:1.8.0">
+        <CLASSES>
+          <root url="jar:///Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/lib/tools.jar!/" />
+        </CLASSES>
+        <JAVADOC />
+        <SOURCES />
+      </library>
+    </orderEntry>
+    <orderEntry type="library" name="Maven: org.slf4j:slf4j-api:1.7.10" level="project" />
+    <orderEntry type="library" name="Maven: org.apache.commons:commons-exec:1.3" level="project" />
+    <orderEntry type="module" module-name="odpswriter" />
+    <orderEntry type="library" name="Maven: com.aliyun.odps:odps-sdk-core:0.19.3-public" level="project" />
+    <orderEntry type="library" name="Maven: com.aliyun.odps:odps-sdk-commons:0.19.3-public" level="project" />
+    <orderEntry type="library" name="Maven: com.jcabi:jcabi-aspects:0.20.1" level="project" />
+    <orderEntry type="library" name="Maven: com.jcabi:jcabi-log:0.15" level="project" />
+    <orderEntry type="library" name="Maven: org.aspectj:aspectjrt:1.8.2" level="project" />
+    <orderEntry type="library" name="Maven: javax.validation:validation-api:1.1.0.Final" level="project" />
+    <orderEntry type="library" name="Maven: org.codehaus.jackson:jackson-mapper-asl:1.9.13" level="project" />
+    <orderEntry type="library" name="Maven: org.codehaus.jackson:jackson-core-asl:1.9.13" level="project" />
+    <orderEntry type="library" name="Maven: net.sourceforge.javacsv:javacsv:2.0" level="project" />
+    <orderEntry type="library" name="Maven: org.bouncycastle:bcprov-jdk15on:1.52" level="project" />
+    <orderEntry type="library" name="Maven: org.xerial.snappy:snappy-java:1.1.1.6" level="project" />
+    <orderEntry type="library" name="Maven: com.google.protobuf:protobuf-java:2.4.1" level="project" />
+    <orderEntry type="library" name="Maven: commons-httpclient:commons-httpclient:3.1" level="project" />
+    <orderEntry type="library" name="Maven: ch.qos.logback:logback-classic:1.0.13" level="project" />
+    <orderEntry type="library" name="Maven: ch.qos.logback:logback-core:1.0.13" level="project" />
+    <orderEntry type="library" name="Maven: mysql:mysql-connector-java:5.1.26" level="project" />
+    <orderEntry type="library" name="Maven: commons-configuration:commons-configuration:1.10" level="project" />
+    <orderEntry type="library" name="Maven: commons-lang:commons-lang:2.6" level="project" />
+    <orderEntry type="library" name="Maven: commons-logging:commons-logging:1.1.1" level="project" />
+    <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
+  </component>
+</module>