我试图实现异步并在spawn子进程中等待。但它不起作用。请看这个
预期产量
*************
http://www.stevecostellolaw.com/
*************
http://www.stevecostellolaw.com/personal-injury.html
http://www.stevecostellolaw.com/personal-injury.html
*************
http://www.stevecostellolaw.com/#
http://www.stevecostellolaw.com/#
*************
http://www.stevecostellolaw.com/home.html
http://www.stevecostellolaw.com/home.html
*************
http://www.stevecostellolaw.com/about-us.html
http://www.stevecostellolaw.com/about-us.html
*************
http://www.stevecostellolaw.com/
http://www.stevecostellolaw.com/
*************
因为每次找到await
生成子对象时,它都会返回python脚本并打印 *************
它,然后打印URL。忽略此处相同url的2次打印
我得到的输出
C:\Users\ASUS\Desktop\searchermc>node app.js
server running on port 3000
DevTools listening on ws://127.0.0.1:52966/devtools/browser/933c20c7-e295-4d84-a4b8-eeb5888ecbbf
[3020:120:0402/105304.190:ERROR:device_event_log_impl.cc(214)] [10:53:04.188] USB: usb_device_handle_win.cc:1056 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[3020:120:0402/105304.190:ERROR:device_event_log_impl.cc(214)] [10:53:04.189] USB: usb_device_handle_win.cc:1056 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
*************
http://www.stevecostellolaw.com/
http://www.stevecostellolaw.com/personal-injury.html
http://www.stevecostellolaw.com/personal-injury.html
http://www.stevecostellolaw.com/#
http://www.stevecostellolaw.com/#
http://www.stevecostellolaw.com/home.html
http://www.stevecostellolaw.com/home.html
http://www.stevecostellolaw.com/about-us.html
http://www.stevecostellolaw.com/about-us.html
http://www.stevecostellolaw.com/
http://www.stevecostellolaw.com/
*************
请参阅下面的app.js
代码
// form submit request
app.post('/formsubmit', function(req, res){
csvData = req.files.csvfile.data.toString('utf8');
filteredArray = cleanArray(csvData.split(/\r?\n/))
csvData = get_array_string(filteredArray)
csvData = csvData.trim()
var keywords = req.body.keywords
keywords = keywords.trim()
// Send request to python script
var spawn = require('child_process').spawn;
var process = spawn('python', ["./webextraction.py", csvData, keywords, req.body.full_search])
var outarr = []
// process.stdout.on('data', (data) => {
// console.log(`stdout: ${data}`);
// });
process.stdout.on('data', async function(data){
console.log("\n ************* ")
console.log(data.toString().trim())
await outarr.push(data.toString().trim())
console.log("\n ************* ")
});
});
Python函数,当if条件匹配时发送URL
# Function for searching keyword start
def search_keyword(href, search_key):
extension_list = ['mp3', 'jpg', 'exe', 'jpeg', 'png', 'pdf', 'vcf']
if(href.split('.')[-1] not in extension_list):
try:
content = selenium_calling(href)
soup = BeautifulSoup(content,'html.parser')
search_string = re.sub("\s+"," ", soup.body.text)
search_string = search_string.lower()
res = [ele for ele in search_key if(ele.lower() in search_string)]
outstr = getstring(res)
outstr = outstr.lstrip(", ")
if(len(res) > 0):
print(href)
found_results.append(href)
href_key_dict[href] = outstr
return 1
else:
notfound_results.append(href)
except Exception as err:
pass
我之所以想这么做,是因为python脚本需要更多的时间来执行,因此每次都会出现超时错误,所以我想在我的nodejs脚本中获得python脚本的中间输出。您可以在下图中看到我遇到的错误
我不确定我是否完全理解你想做什么,但我会试一试,因为你似乎已经问了很多次这个问题(这通常不是一个好主意)。我认为你的问题不够清晰——如果你能澄清你的最终目标是什么(即,你希望它如何运作?),这会有很大帮助
我想你在这里提到了两个不同的问题。首先,您希望在从脚本返回的每个单独的数据之前放置一行新的“******”。这是一个不可依赖的问题——请查看此问题的答案以了解更多详细信息:Order of process.stdout.on( 'data', ... ) and process.stderr.on( 'data', ... )。数据将以块的形式传递给stdout处理程序,而不是逐行传递,根据管道中当前的数据量,可以一次提供任意数量的数据
我最困惑的部分是您的措辞“在我的nodejs脚本中获得python脚本的中间输出”。不一定有任何“即时”数据——您不能依赖于进程的标准输出处理程序在任何特定时间输入的数据,它将以Python脚本本身及其运行的进程确定的速度向您提供数据。话虽如此,听起来你的主要问题是你的帖子超时了。你永远不会结束你的请求-这就是为什么你会得到一个超时。我将假设您希望在发送回响应之前等待第一个数据块(不管它包含多少行)。在这种情况下,您需要添加res.send,如下所示:
相关问题 更多 >
编程相关推荐