jttp

NAME

jttp - Send an HTTP(S) request to a web server and process the response.

BUILDING

Install Apache Maven and run mvn clean package. This will execute the tests and produce the jttp.jar binary.

SYNOPSIS

java [jvm args] -jar /path/to/jttp.jar [-dhNORvV] [-A user[:password]] [-M mimetype] [-o filename] [-p
            entity] [-P NONE|COLOR|INDENT|ALL] [--post-process-script-name
            scriptname] [--pre-process-script-name scriptname] [-S sessionname]
            [-X methodname] [--post-process-script-arg arg]...
            [--pre-process-script-arg arg]... [@<filename>...] url
            [request_item...]

DESCRIPTION

Command line HTTP(S) client, similar to httpie. Offers numerous input and output methods and many authentication options. Input for forms or json can be set as arguments from the command line or as redirected input files. Simple json is generated from command line arguments. For more complex formats, use input redirection.

Jttp may be extended or enhanced by using scripts that execute before and after the request is made. These scripts may be written in Groovy.

ARGUMENTS

url

Request url. Scheme defaults to http:// if the URL doesn't include one. You can also use a shorthand for localhost

      $ java Jttp.java :8080     # => http://localhost:8080
      $ java Jttp.java /foo      # => http://localhost/foo
  

request_item

Optional key-value pairs to be included in the request. The separator used determins the type:

':' HTTP headers:

Referer:http://example.com Cookie:foo=bar User-Agent:Crawler/1.0

'==' URL parameters to be appended to the URI (query string):

el==toro marco==polo

'=' Data fields to be serialized into a JSON object (with -M JSON) or form data (with -M FORM):

fizz=buzz bar=baz foo=fu

'@' Form file fields:

filename@Documents/report.docx

OPTIONS

-A,--auth user[:passwd]

If only the username is provided, (e.g. -A user), Jttp will prompt for the password.

-d,--download

Do not print the response body to stdout. Rather, download it and store it in a file. The filename is guessed unless specified with -o filename. If the value of -o is not an absolute path, the output is saved in a relative directory to the current directory. If nothing is specified and no preference for download directory is set (see PREFERENCES), then the file is saved to

  ${user.home}/.jttp/downloads/<GUESSED-FILENAME>

-h,--help

Shows a detailed help message and exits.

-M,--request-mime-type mimetype

Set the request MIME type. Should be one of FORM, JSON or MULTIPART.

-N,--no-verify

Turn off certificate and host name checking. The internal logger will emit a WARNING message when this option is set.

-O,--offline

Build the request and print it, but don't actually send it.

-o,--output filename

Save output to filename instead of stdout. If -d is also set, then only the response body is saved to filename.

-P,--pretty-print NONE|COLOR|INDENT|ALL

Controls output processing. The value can be NONE to not prettify the output, ALL to apply both colors and indenting (default when printing to System.out), COLOR, or INDENT.

--post-process-script script_name

Script to run after the request has fetched data but before final output is handled by Jttp.

--post-process-script-arg script_arg

Argument to pass to the --post-process-script. This can be specified as many times as you need for your script. The args will be passed to the script as an array of Strings to the script in a variable bound to the name args.

--pre-process-script

Script to run before the request has fetched data but after initial setup has been performed by Jttp.

--pre-process-script-arg

Argument to pass to the --pre-process-script. This can be specified as many times as you need for your script. The args will be passed to the script as an array of Strings to the script in a variable bound to the name args.

-p,--print what

String specifying what the output should contain:

"H"

Request headers

"B"

Request body

"h"

Resposne headers

"b"

Response body

The default behavior is "hb" (the response headers and body is printed).

-R,--read-only-session

Load the named session, but don't change it when processing the response. Ignored if -S sessionname isn't specified.

-S,--session sessionname

Create or reuse and update a session. Within a session, headers, request history, cookies and response data are persisted between requests.

By default, session files are stored in:

  ${user.home}/.jttp/sessions/<HOST>/<SESSION_NAME>.zip

The zip file contains the files: headers.xml, history.xml, any response data, and cookies.xml.

-v,--verbose

Print request and response headers and body. Shortcut for -p HBhb.

-V,--version

Shows the version information and exits.

-X,--request-method methodname

HTTP method to run, one of DELETE, GET (default), HEAD, OPTIONS, POST, PUT, or TRACE.

EXIT STATUS

0

Successful run.

1

One of the following occurred:

• A required option was not set.
• An option with a required argument was missing its argument.
• An exception was thrown/raised.

2

Either no options were set or the --help or --version option was set.

FILES

messages_jttp.properties

Contains all application messages. Useful for i18n messages for this application. This file needs to be in the same directory as Jttp.java.

${user.home}/.jttp/downloads

Default directory for storing downloaded responses.

${user.home}/.jttp/scripts

Default directory for storing scripts.

${user.home}/.jttp/sessions

Default directory for storing session data.

PREFERENCES

Jttp does not add preferences when it runs. You should use another tool to add these preferences under the user root with the node name /com/github/argherna/jttp/<subnode>. The last qualifier in the preferences documented is the preference name.

user/com/github/argherna/jttp/directories/base

Base directory for most save information (default is ${user.home}/.jttp).

user/com/github/argherna/jttp/directories/downloads

Absolute pathname for storing downloads (from using the --download option with no --output option set; default is ${base}/downloads where ${base} is the value of the base preference above).

user/com/github/argherna/jttp/directories/scripts

Directory (under ${base}) where scripts are stored (default is ${base}/scripts).

user/com/github/argherna/jttp/directories/sessions

Directory (under ${base}) where session files are stored (default is ${base}/sessions).

SYSTEM PROPERTIES

Jttp is implemented with the HttpURLConnection. This means all system properties that apply to it are applicable to Jttp with all desired side-effects.

The Jttp-specific system properties that can be set are:

jttp.indent

Number of spaces to indent output when formatting (default is 2).

jttp.keep.tempfiles

When true, don't delete any temporary files produced by the run (default is false).

Other system properties are also used by internal subsystems:

content.types.user.table

Path to a custom filename map (mimetable), used by the java.net.FileNameMap implementation to determine file mimetype by looking at the file extension.

http.auth.preference

One of SPNEGO, digest, NTLM, or basic (which is the default). See the Http Authentication reference below for more information.

java.security.auth.login.config

Path to the JAAS configuration file.

java.security.krb5.conf

Path to the Kerberos configuration file. Ignored unless SPNEGO authentication is specified.

java.util.logging.config.file

Standard Java System Property that points to a logging configuration file.

javax.net.debug

Write SSL debug information to System.err. See the references below related to debugging SSL.

javax.net.ssl.keyStore

Standard Java System Property that points to the keystore to use for client authentication.

javax.net.ssl.keyStorePassword

Standard Java System Property that is the password for the keystore specified by javax.net.ssl.keyStore.

javax.net.ssl.keyStoreType

Standard Java System Property that is the type for the keystore specified by javax.net.ssl.keyStore (default is JKS).

javax.net.ssl.trustStore

Standard Java System Property that points to the truststore to use for server authentication (default is the JRE's cacerts file).

javax.net.ssl.trustStorePassword

Standard Java System Property that is the password for the truststore specified by javax.net.ssl.trustStore (default is changeit).

javax.net.ssl.trustStoreType

Standard Java System Property that is the type for the truststore specified by javax.net.ssl.trustStore (default is JKS).

javax.security.auth.useSubjectCredsOnly

Set to false to make the underlying Kerberos mechanism obtain Kerberos credentials. Jttp does not support a setting of true.

sun.security.spnego.debug

Writes SPNEGO debug messages to System.err. Useful for troubleshooting SPNEGO authentication issues.

sun.security.krb5.debug

Writes Kerberos debug messages to System.err. Useful for troubleshooting SPNEGO authentication issues.

EXAMPLES

Examples are formatted using unix multiline style.

Common Uses

Generate the request but don't send it.

java -jar jttp.jar -Ov http://www.example.com

Save session information (useful to keep a cookie-based session going after login).

java -jar jttp.jar \
  -S my_session \
  https://www.example.com/secure

Send a form with a file to upload.

java -jar jttp.jar \
  -M FORM POST \
  https://www.example.com/expenses \
  user=myUserId \
  statement@Documents/expenses.xlsx

Set a custom header.

java -jar jttp.jar \
  http://www.example.com/ua-sniffer \
  User-Agent:my-custom-ua/1.2.3.4

Using System Properties

Note that you can set system properties in a text file (for example call it argsfile), one per line defined as -Dname=value as usual and then pass the filename to the java command with @argsfile. This is a useful shorthand for making repeated requests using the same settings. For illustrative purposes, these examples will specify the system properties on the command line.

Use SPNEGO authentication. Assume a krb5.conf file and JAAS configuation file my-jaas.conf exist at the specified locations.

java -Dhttp.auth.preference=SPNEGO \
  -Djava.security.auth.login.config=/secure/my-jaas.conf \
  -Djava.security.krb5.conf=/secure/krb5.conf \
  -Djavax.security.auth.useSubjectCredsOnly=false \
  -jar jttp.jar https://www.example.com/SPNEGO/doc

Use your keystore to authenticate your client (note the Url for cryptomix.com is a free service to test client authentication).

java -Djavax.net.ssl.keyStore=/home/user/.keystore \
  -Djavax.net.ssl.keyStorePassword=changeit \
  -jar jttp.jar https://server.cryptomix.com/secure/

Scripts

You can specify scripts to run before and after the actual HTTP request is run. The Groovy scripting language jars are part of the jttp.jar file. If you specify the same script name for both --pre-process-script and --post-process-script, they are executed separately using separate instances of the underlying Java scripting objects. This example shows how to tell Jttp to run a script called pre-auth.groovy before running the HTTP request. You can specify scripts written with different languages for both scripts.

java -jar jttp.jar \
  --pre-process-script pre-auth.groovy \
  https://authed.example.com/secure

You can specify options to pass to the --pre or --post processing scripts.

java -jar jttp.jar \
  --pre-process-script pre-auth.groovy \
  --pre-process-script-arg "--userid bob" \
  --pre-process-script-arg "--keystore /home/bob/.keystore" \
  https://authed.example.com/secure

And you can specify scripts to run both before and after the HTTP request has been made.

java -jar jttp.jar \
  --pre-process-script pre-auth.groovy \
  --pre-process-script-arg "--userid bob" \
  --pre-process-script-arg "--keystore /home/bob/.keystore" \
  --post-process-script post-dbload.groovy \
  --post-process-script-arg "--jdbcurl jdbc:hsqldb:mem:mydb" \
  --post-process-script-arg "--jdbcuser sa" \
  --post-process-script-arg an_arg \
  https://authed.example.com/secure

NOTES

Jttp was inspired by and has several features and options copied from httpie. The motivation for Jttp was to implement a client using the venerable HttpURLConnection to demonstrate how to use it effectively. The HttpURLConnection is versatile and takes advantage of many built-in features to the Java Runtime through system properties including SPNEGO authentication, client certificate authentication, basic authentication, and custom MIME-type databases. Since the intended use for Jttp is as a command line program, system properties are well-suited for this purpose. StackOverflow provides an excellent essay on how to use HttpURLConnection the way it was intended. This shows that the API does not necessarily have to mirror the actual conversation that takes place between clients and servers but rather acts as a way to send and receive data to process.

Response Body Processing

Jttp will download all responses to the java.io.tmpdir location (usually $TMPDIR). It will then either read from the file locally to produce output that can be formatted for indentation and color or copy the file to the downloads directory. The temporary files are deleted at the end of the run unless the jttp.keep.tempfiles system property is specified with a value of true. Keeping the temporary files is useful for debugging certain issues that can arise during execution. Otherwise they should just be thrown away.

Scripting

Jttp binds an object to the scripting engine called jttpScriptObject that gives scripts access to:

• The HttpURLConnection (through the getHttpURLConnection method) in the current state it's in depending on when the script was executed. If it is executed before the HTTP request is run, you can set headers, etc. You don't have to set the method since Jttp does that for you. If it is executed after the HTTP request is run, you have access to the response headers. It's best practice to use scripting to set headers before the HTTP request is run. It's best practice to not modify any responses or response data in the HttpURLConnection after the HTTP request is run.
• The temporary response file (through the getResponseFile method), but only in scripts run after the HTTP request is run. It's best practice to use the temporary file to read and process but not modify.
• A logger (through the log method) that will write log messages to the logger used by Jttp at INFO level.

@-files and Shell Aliases

Both the java launcher and Jttp recognize at-files (@-files). These are files that you can make that contains either a jvm or program argument (1 per line). Combining with shell aliases, this makes jttp incredibly easy to use. For example, if you have to make repeated requests to a server and you need to take advantage of many of the system properties on the jvm, make a file called /home/user/jttp-args and give it the following:

-Dhttp.auth.preference=SPNEGO
-Djava.security.auth.login.config=/secure/my-jaas.conf
-Djava.security.krb5.conf=/secure/krb5.conf
-Djavax.security.auth.useSubjectCredsOnly=false
-jar
/path/to/jttp

Now run jttp:

java @/home/user/jttp-args https://spnego.example.com/secure

You can edit /home/user/jttp-args to have whatever you want in it. Similarly, you could make an @-file for jttp args, but that has to be a separate file. For example, make a file called /home/user/jttp-pgm-args and give it the following:

-v
-X POST
-M JSON
https://httpbin.org/post

Now run jttp and redirect a file called /home/user/httpbin-post.json as input:

java @/home/user/jttp-args @/home/user/jttp-pgm-args < /home/user/httpbin-post.json

Finally, you could define a shell alias to make this go even faster. For example, this works in bash and zsh:

alias jttp="java @/home/user/jttp-args"

Now you can simply run jttp as follows:

jttp @/home/user/jttp-pgm-args < /home/user/httpbin-post.json

You can combine aliases as well in any way you want to make your life easier when running jttp.

SEE ALSO

• httpie
• Http Authentication
• (Kerberos) Troubleshooting
• Java Logging Configuration
• Debugging SSL/TLS Connections
• (SSL) Debugging Utilities
• How to use java.net.URLConnection to fire and handle HTTP requests?
• Package javax.script

AUTHOR

Andy Gherna <mailto: [email protected]>

ISSUES

Reporting

Report issues at https://github.com/argherna/jttp/issues.

Known Issues

• Support for the PATCH method is server dependent.
• HTTP/2 is not supported.
• Asynchronous requests are not supported.
• Rendering issues will occur when --pretty-print INDENT or --pretty-print ALL or the default settings are set depending on the document received. This is due to the formatting done by Jttp and can be adjusted (you have the code in front of you and I love reviewing pull requests).