current position:Home>Sorting out the knowledge blind area of Java crawler
Sorting out the knowledge blind area of Java crawler
2022-01-27 04:15:37 【Big flicker love flicker】
java Sorting out the blind area of reptile knowledge
HttpClient Redirection processing
【HttpClient4.5 Chinese Course 】 8、 ... and . Terminate request and redirect processing
First, HttpClient And the difference between browsers
We make a request from the browser , The browser will help you handle redirection 、 Cache and other things . That's why you use browser forms post After submission , No matter how the server redirects , Can normally receive the data returned by the server .
But with HttpClient Well , You'll find that , After the request , Returns the 302, because POST Mode submission HttpClient It won't help you deal with redirection . What do you do then ?
Method 1 :( Handle it manually )
HttpClient httpClient = HttpClients.createDefault();
HttpPost httpPost= new HttpPost(http://ip:port/xxx);
CloseableHttpResponse response = httpclient.execute(httpPost);
int statusCode = response.getStatusLine().getStatusCode();
System.out.println("statusCode=="+statusCode); // Return code
Header header=response.getFirstHeader("Location");
// Redirect address
String location = header.getValue();
System.out.println(location);
// And then on the new location Just make a request
HttpGet httpGet = new HttpGet(location);
CloseableHttpResponse response2 = httpclient.execute(httpGet);
System.out.println(" Return message "+EntityUtils.toString(response2.getEntity(), "UT-F-8"));
Method 2 :( Existing tool classes )
HttpClientBuilder builder = HttpClients.custom()
.disableAutomaticRetries() // Turn off automatic processing redirection
.setRedirectStrategy(new LaxRedirectStrategy());// utilize LaxRedirectStrategy Handle POST Redirection problem
CloseableHttpClient client = builder.build();
HttpPost httpPost= new HttpPost(http://ip:port/xxx);
CloseableHttpResponse response = client.execute(httpPost);
int statusCode = response.getStatusLine().getStatusCode();
System.out.println("statusCode=="+statusCode); // Return code
System.out.println(" Return message "+EntityUtils.toString(response.getEntity(), "UT-F-8"));
HttpClient obtain Cookie Two ways
One 、 The old version of HttpClient obtain Cookies
p.s. This method is officially not recommended
Use DefaultHttpClient Class instantiation httpClient object :
public static String dooPost_deprecated(String url, Map<String, String> map, String charset) {
DefaultHttpClient httpClient = null;
HttpPost httpPost = null;
String result = null;
try {
httpClient = new DefaultHttpClient();
httpPost = new HttpPost(url);
// Set parameters
List<NameValuePair> list = new ArrayList<NameValuePair>();
Iterator<Entry<String, String>> iterator = map.entrySet().iterator();
while (iterator.hasNext()) {
Entry<String, String> elem = (Entry<String, String>) iterator.next();
list.add(new BasicNameValuePair(elem.getKey(), elem.getValue()));
}
if (list.size() > 0) {
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(list, charset);
httpPost.setEntity(entity);
}
HttpResponse response = httpClient.execute(httpPost);
System.out.println(response.getStatusLine().getStatusCode());
String JSESSIONID = null;
String cookie_user = null;
// get Cookies
CookieStore cookieStore = httpClient.getCookieStore();
List<Cookie> cookies = cookieStore.getCookies();
for (int i = 0; i < cookies.size(); i++) {
// Traverse Cookies
System.out.println(cookies.get(i));
System.out.println("cookiename=="+cookies.get(i).getName());
System.out.println("cookieValue=="+cookies.get(i).getValue());
System.out.println("Domain=="+cookies.get(i).getDomain());
System.out.println("Path=="+cookies.get(i).getPath());
System.out.println("Version=="+cookies.get(i).getVersion());
if (cookies.get(i).getName().equals("JSESSIONID")) {
JSESSIONID = cookies.get(i).getValue();
}
if (cookies.get(i).getName().equals("cookie_user")) {
cookie_user = cookies.get(i).getValue();
}
}
if (cookie_user != null) {
result = JSESSIONID;
}
} catch (Exception ex) {
ex.printStackTrace();
}
return result;
}
Two 、 The new version of the HttpClient obtain Cookies
Use CloseableHttpClient Class instantiation httpClient object :
public static String doPost(Map<String, String> map, String charset) {
CloseableHttpClient httpClient = null;
HttpPost httpPost = null;
String result = null;
try {
CookieStore cookieStore = new BasicCookieStore();
httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build();
httpPost = new HttpPost("http://localhost:8080/testtoolmanagement/LoginServlet");
List<NameValuePair> list = new ArrayList<NameValuePair>();
Iterator<Map.Entry<String, String>> iterator = map.entrySet().iterator();
while (iterator.hasNext()) {
Entry<String, String> elem = (Entry<String, String>) iterator.next();
list.add(new BasicNameValuePair(elem.getKey(), elem.getValue()));
}
if (list.size() > 0) {
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(list, charset);
httpPost.setEntity(entity);
}
httpClient.execute(httpPost);
String JSESSIONID = null;
String cookie_user = null;
List<Cookie> cookies = cookieStore.getCookies();
for (int i = 0; i < cookies.size(); i++) {
if (cookies.get(i).getName().equals("JSESSIONID")) {
JSESSIONID = cookies.get(i).getValue();
}
if (cookies.get(i).getName().equals("cookie_user")) {
cookie_user = cookies.get(i).getValue();
}
}
if (cookie_user != null) {
result = JSESSIONID;
}
} catch (Exception ex) {
ex.printStackTrace();
}
return result;
}
copyright notice
author[Big flicker love flicker],Please bring the original link to reprint, thank you.
https://en.cdmana.com/2022/01/202201270415324929.html
The sidebar is recommended
- Spring IOC container loading process
- [thinking] the difference between singleton mode and static method - object-oriented programming
- Hadoop environment setup (MySQL environment configuration)
- 10 minutes, using node JS creates a real-time early warning system for bad weather!
- Git tool
- Force deduction algorithm - 92 Reverse linked list II
- What is the sub problem of dynamic programming?
- C / C + +: static keyword summary
- Idea does not have the artifacts option when configuring Tomcat
- Anaconda can't open it
guess what you like
-
I don't know how to start this
-
Matlab simulation of transportation optimization algorithm based on PSO
-
MySQL slow log optimization
-
[Vue] as the window is stretched (larger, smaller, wider and higher), the text will not be displayed
-
Popular Linux distributions for embedded computing
-
Suzhou computer research
-
After installing SSL Certificate in Windows + tomcat, the domain name request is not successful. Please answer!!
-
Implementation time output and greetings of jQuery instance
-
The 72 year old uncle became popular. Wu Jing and Guo fan made his story into a film, which made countless dreamers blush
-
How to save computer research
Random recommended
- Springboot implements excel import and export, which is easy to use, and poi can be thrown away
- The final examination subjects of a class are mathematical programming, and the scores are sorted and output from high to low
- Two pronged approach, Tsinghua Professor Pro code JDK and hotspot source code notes, one-time learning to understand
- C + + recursive knapsack problem
- The use of GIT and GitHub and the latest git tutorial are easy to understand -- Video notes of crazy God speaking
- PostgreSQL statement query
- Ignition database test
- Context didn't understand why he got a high salary?, Nginxfair principle
- Bootstrap switch switch control user's guide, springcloud actual combat video
- A list that contains only strings. What other search methods can be used except sequential search
- [matlab path planning] multi ant colony algorithm grid map path planning [including GUI source code 650]
- [matlab path planning] improved genetic algorithm grid map path planning [including source code phase 525]
- Iinternet network path management system
- Appium settings app is not running after 5000ms
- Reactnative foundation - 07 (background image, status bar, statusbar)
- Reactnative foundation - 04 (custom rpx)
- If you want an embedded database (H2, hsql or Derby), please put it on the classpath
- When using stm32g070 Hal library, if you want to write to flash, you must perform an erase. If you don't let it, you can't write continuously.
- Linux checks where the software is installed and what files are installed
- SQL statement fuzzy query and time interval filtering
- 69. Sqrt (x) (c + + problem solving version with vs runnable source program)
- Fresh students are about to graduate. Do you choose Java development or big data?
- Java project: OA management system (java + SSM + bootstrap + MySQL + JSP)
- Titanic passenger survival prediction
- Vectorization of deep learning formula
- Configuration and use of private image warehouse of microservice architect docker
- Relearn JavaScript events
- For someone, delete return 1 and return 0
- How does Java dynamically obtain what type of data is passed? It is used to judge whether the data is the same, dynamic data type
- How does the database cow optimize SQL?
- [data structure] chain structure of binary tree (pre order traversal) (middle order traversal) (post order traversal) (sequence traversal)
- Webpack packaging optimization solution
- 5. Operation element
- Detailed explanation of red and black trees
- redhat7. 9 install database 19C
- Blue Bridge Cup notes: (the given elements are not repeated) complete arrangement (arrangement cannot be repeated, arrangement can be repeated)
- Detailed explanation of springboot default package scanning mechanism and @ componentscan specified scanning path
- How to solve the run-time exception of test times
- Detailed explanation of k8s management tool kubectl
- Android system view memory command