Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some updates to the Kinesis patch #1

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Some updates to the Kinesis patch #1

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
b3b0ff1
[SPARK-1981] Add AWS Kinesis streaming support
cfregly Jul 16, 2014
d18e680
Merge remote-tracking branch 'upstream/master'
cfregly Jul 16, 2014
cd68c0d
fixed typos and backward compatibility
cfregly Jul 19, 2014
e7c8978
Merge remote-tracking branch 'upstream/master'
cfregly Jul 22, 2014
828f8ae
more cleanup
cfregly Jul 22, 2014
338997e
improve build docs for kinesis
cfregly Jul 22, 2014
6c39561
parameterized the versions of the aws java sdk and kinesis client
cfregly Jul 24, 2014
21de67f
Merge remote-tracking branch 'upstream/master'
cfregly Jul 31, 2014
db3eefd
Merge remote-tracking branch 'upstream/master'
cfregly Jul 31, 2014
912640c
changed the foundKinesis class to be a publically-avail class
cfregly Jul 31, 2014
fd8dce4
Minified the KinesisWordCount example.
tdas Jul 31, 2014
47a30a9
Fixed scala docs of KinesisUtils
tdas Aug 1, 2014
a9811b7
Fixed mistakes.
tdas Aug 1, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Fixed mistakes.
  • Loading branch information
@tdas
tdas committed Aug 1, 2014
commit a9811b7a93ef047faed685373a9107a830e9f88a
8 changes: 4 additions & 4 deletions extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ object KinesisUtils extends Logging {
checkpointIntervalMillis: Long,
initialPositionInStream: InitialPositionInStream,
storageLevel: StorageLevel): ReceiverInputDStream[Array[Byte]] = {
ssc.receiverStream(new KinesisReceiver(appName, streamName, endpointUrl, checkpointIntervalMillis,
initialPositionInStream, storageLevel))
ssc.receiverStream(new KinesisReceiver(appName, streamName, endpointUrl,
checkpointIntervalMillis, initialPositionInStream, storageLevel))
}

/**
Expand Down Expand Up @@ -100,7 +100,7 @@ object KinesisUtils extends Logging {
checkpointIntervalMillis: Long,
initialPositionInStream: InitialPositionInStream,
storageLevel: StorageLevel): JavaReceiverInputDStream[Array[Byte]] = {
jssc.receiverStream(new KinesisReceiver(appName, stream, endpoint, checkpointIntervalMillis,
initialPositionInStream, storageLevel))
createStream(jssc.ssc, appName, streamName, endpointUrl,
checkpointIntervalMillis, initialPositionInStream, storageLevel)
}
}