OkHttp3 Scripting
Postman can help you generate http call code in Java, curl and lots of other languages
It´s a very fast way to do some scripting with Java, suited for requirements of batching nature
One use case I came across was to fire 50K+ HTTP calls, reading input from a csv.
Granted, these use cases are more suited for SQL, PL/SQL in my experience, but you don´t always have access to the persistence layer.
Grab Example
Let´s test with some public APIs:
using urlHaus for instance: https://urlhaus-api.abuse.ch/v1/urls/recent/
Dependencies
add okhttp3 and commons-lang3 for the wonderful ExceptionUtils:
pom.xml
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>4.12.0</version>
</dependency>
<!-- optional -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
Gift from Postman
Grab the snippet from the right panel:
OkHttpClient client = new OkHttpClient().newBuilder()
.build();
MediaType mediaType = MediaType.parse("text/plain");
RequestBody body = RequestBody.create(mediaType, "");
Request request = new Request.Builder()
.url("https://urlhaus-api.abuse.ch/v1/urls/recent/")
.method("GET", body)
.build();
Response response = client.newCall(request).execute();
Files.lines and start some fire ( on your own backend :D )
Processing a file in the form A;B;C where we only need use B and C
just to explain the splits
Input example:
xxxx;B0;C0
xxxx;B1;C1
xxxx;B2;C2
xxxx;B3;C3
//to append to output file names
//report successes in success-<timestamp>.txt and failures in failure-<date>.txt
//to distinguish multiple runs
LocalDateTime ldt = LocalDateTime.now();
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd--HH-mm-ss");
public void batch() throws IOException, URISyntaxException {
Files.lines(Path.of(getClass().getResource("/input.txt").toURI()))
.parallel()
.forEach(
l -> {
try {
this.process(l.split(";")[1], l.split(";")[2]);
} catch (URISyntaxException | IOException ex) {
log.info("io exception trace " + ExceptionUtils.getStackTrace(ex));
ex.printStackTrace();
}
}
);
}
void process(String x,String y) method:
log.info("processing x " + x + " y " + y);
try {
OkHttpClient client = new OkHttpClient().newBuilder()
.build();
MediaType mediaType = MediaType.parse("text/plain");
RequestBody body = RequestBody.create(mediaType, "");
Request request = new Request.Builder()
.url("https://urlhaus-api.abuse.ch/v1/urls/recent/")
.method("GET", body)
.addHeader("x", x)
.addHeader("y", y)
.build();
try (Response response = client.newCall(request).execute()) {
log.info(" current response " + response);
String respBody = response.body() == null ? "" : response.body().string();
if (response.isSuccessful()) {
String str = "xxxx" + ";" + x + ";" + y + ";" + response + ";" + respBody + System.lineSeparator();
log.info("operation succeded for " + x + " y " + y);
Files.writeString(Paths.get("logs", "success-" + formatter.format(ldt) + ".txt"), str, StandardOpenOption.APPEND, StandardOpenOption.CREATE);
} else {
String str = "xxxx" + ";" + x + ";" + y + ";" + response + ";" + respBody + System.lineSeparator();
log.info("operation failed for " + x + " y " + y);
Files.writeString(Paths.get("logs", "failure-" + formatter.format(ldt) + ".txt"), str, StandardOpenOption.APPEND, StandardOpenOption.CREATE);
}
}
} catch (Exception ex) {
log.info("operation failed for " + x + " y " + y + " exception " + ex.getMessage());
log.info("exception trace " + ExceptionUtils.getStackTrace(ex));
String str = "xxxx" + ";" + x + ";" + y + ";" + ex.getMessage() + System.lineSeparator();
Files.writeString(Paths.get("logs", "failure-" + formatter.format(ldt) + "-" + ".txt"), str, StandardOpenOption.APPEND, StandardOpenOption.CREATE);
ex.printStackTrace();
}
Calculate your QPS
Now that you can fire queries as many as there are cores in your processors on your laptop
time for some quick math:
![Quick maths mans not hot]()
Let´s assume :
you will be sending 10 simultaneous queries that will last 400ms each
this means each thread in the common thread pool will handle 2.5 (1s/400ms) queries per second
which means you will be sending queries roughly at the rate of 2.5*10(cores) = 25 QPS (query per second)
Make sure your api gateway, your backend, and the max QPS for your api key can handle that load
You can roughly process the 50K with that rate in :
50K total queries / 25 QPS = 2000s ~ 34 mins
Last modified: 12 March 2024
