(CORE) CORE (2022)

Asynchronous MultiThreaded SSH engine for Web (Net Core 6, Linux) - Part 1,2 (Database and SSH client). StreamReader/ReadToEndAsync, Extension/Inherits/Overloads, BeginExecute/EndExecute/IAsyncResult/IProgress(Of T)/Async/Await/CancellationToken/Task.Run/Thread.Yield.

I like write various Asynchronous MultiThreaded engine, look for example this my posts.

And whole my life I created various implementation of PUSH notification from server to client, look for example this my post.

Of course, if you use MsSQL server, the best way is using this future Use SqlDependency/SqlServiceBroker to send notification to SignalR hub, but similar future is absent in other SQL Server, this is, of course, is hard proof why EF Code First is absolutely idiotic idea, because there nothing similar function in different SQL Server (except store data to tables). At common EF Code First has so many other restrictions, for example this engine even don't understand what is View, impossible to work with existing database (EF DB First theoretically allow working, but in practice there are nothing EF providers for other DB, except DB from Microsoft). Therefore, if you want to use similar futures like SqlDependency/SqlServiceBrokerSqlDependency/SqlServiceBroker for other DB, you need to realize own engine. But disadvantage of EF is another story - Класифікація засобів роботи з даними.Класифікація засобів роботи з даними, in next parts I will describe my way to access Data more details.

This Engine I created as evolution and developing of two projects I described so far in my blog:


Engine I described in this page allow to order any SSH command to Queue to each VM and Servers, than perform each SSH command one-by-one for each BareMetal Server or each project VM, some command can working at many hours, for example DD like this, and than client receive notification with result by SignalR. In the screen below you can see how this engine ordered SSH command to a Queue for a couple VM and Servers, than perform it one-by-one, and send result to backend.



1. Database.

So, The first part of this Engine is Database MySQL Kvm hosting management Db with tracking changing by triggers. This Database is huge and not all tables is need to this Engine, this Engine use only one table

Function of this Engine also related with table User (this is reference to ToUser), table VM (this is reference ToVm of Virtual Machines of project) and table KvmHost (this is reference ToServer of Bare Metal Server of project). Password to access for Server or VM will be produced from fiels SshDecryptPass. I hope reader understanding goal of other fields.

For Access with MySQL I use this way. I will write more details about masking real password to Db later. This is one of this keys to protect project from rogue intrusion.



My project use this providers to access MySQL (MariaDB).



2. SSH client.

Next component of this Engine is Asynchronous SSH client. As access to bash I use https://github.com/sshnet/SSH.NET/, but this is only one line of code in this library. I wrap SSH engine to separate library and uploaded this component to Github https://github.com/Alex-1557/AsyncSshEngine.

This component composed from six classes/modules.



This library refs to two POCO-classes with describe information about BareMetal Servers and Virtual Machines. And final classes separated to two namespaces - related to Vm and related to Servers.



Also this library used two Net core service ApplicationDbContext and AesCryptor, I will describe this services in the next pages.


So, now we look to this library. ServerBash is my base class, when it creating it reading information from DB about server it describe. Similar class is exists in VM namespace with little VM specific.

In derived class VmBashAsync I redefine main method Bash from base class ServerBash with Asynchronous pattern. In this class I use my own extension ExecuteAsync, system class Progress, and class to collect output of SSH.NET SshOutputLine.



   1:  Imports BackendAPI.Model
   2:  Imports BackendAPI.Services
   3:   
   4:  Namespace Kvm
   5:      Public Class ServerBashAsync
   6:          Inherits ServerBash
   7:   
   8:          Public SshOutput As List(Of String)
   9:          Public SshErrMsg As List(Of String)
  10:          Public Sub New(_DB As ApplicationDbContext, _Aes As AesCryptor, ServerI As Integer, ServerDecryptPass As String)
  11:              MyBase.New(_DB, _Aes, ServerI, ServerDecryptPass)
  12:              SshErrMsg = New List(Of String)
  13:              SshOutput = New List(Of String)
  14:          End Sub
  15:   
  16:          Public Overloads Async Function Bash(BashCmd As String) As Task(Of String)
  17:              Dim CTX = New Threading.CancellationTokenSource()
  18:              If SshClient.IsConnected Then
  19:                  Try
  20:                      Dim Cmd1 = SshClient.CreateCommand(BashCmd)
  21:                      Await Task.Run(Function() Cmd1.ExecuteAsync(New Progress(Of SshOutputLine), CTX.Token, SshOutput, SshErrMsg, KvmHosts.MainServerIP))
  22:                      'Await Cmd1.ExecuteAsync(New Progress(Of SshOutputLine), CTX)
  23:                  Catch ex As Exception
  24:                      Return ex.Message
  25:                  End Try
  26:              Else
  27:                  Await Task.Run(Function() "not connected")
  28:              End If
  29:          End Function
  30:   
  31:      End Class
  32:  End Namespace

SshOutputLine is a simple collector of output with divide Error and Normal output to two class (debugging is commented, it can show to application console each portion of information receiving from BASH interpreter).



   1:  Public Class SshOutputLine
   2:      Public Shared Property LineCount As Integer
   3:   
   4:      Public Sub New(ByVal Line As String, ByVal IsErrorLine As Boolean, SshOutput As List(Of String), SshErrMsg As List(Of String), IpAddr As String)
   5:          'Debug.WriteLine($"Line Ready ({Now}): {Line}")
   6:          LineCount += 1
   7:          Debug.WriteLine($"{IpAddr} Line Ready ({Now}): {LineCount}")
   8:   
   9:          If IsErrorLine Then
  10:              SshErrMsg.Add(Line)
  11:          Else
  12:              SshOutput.Add(Line)
  13:          End If
  14:      End Sub
  15:  End Class

If we uncomment debugging we will see LineReady message - order where each line will from Bash interpreter will be ready.



And last component is module with Async extension SshCommandExtensions.



   1:  Imports System.IO
   2:  Imports System.Runtime.CompilerServices
   3:  Imports System.Threading
   4:  Imports Renci.SshNet
   5:   
   6:  Module SshCommandExtensions
   7:      <Extension()>
   8:      Async Function ExecuteAsync(ByVal SshCommand As SshCommand, ByVal OutputLine As IProgress(Of SshOutputLine), ByVal CTX As CancellationToken, SshOutput As List(Of String), SshErrMsg As List(Of String), IpAddr As String) As Task
   9:          Dim AsyncResult As IAsyncResult = SshCommand.BeginExecute()
  10:          Dim StdoutSR = New StreamReader(SshCommand.OutputStream)
  11:          Dim StderrSR = New StreamReader(SshCommand.ExtendedOutputStream)
  12:          While Not AsyncResult.IsCompleted
  13:              'Debug.Print($"#1 {AsyncResult.IsCompleted} {Now} {String.Join(".", SshOutput)}")
  14:              ' Debug.Print($"#1 {IpAddr} {AsyncResult.IsCompleted} {Now} ")
  15:              Await Progress(SshCommand, StdoutSR, StderrSR, OutputLine, CTX, SshOutput, SshErrMsg, IpAddr)
  16:              Thread.Yield()
  17:              'Debug.Print($"#2 {AsyncResult.IsCompleted} {Now} {String.Join(".", SshOutput)}")
  18:              'Debug.Print($"#2 {IpAddr} {AsyncResult.IsCompleted} {Now} ")
  19:          End While
  20:          'Debug.Print($"#3 {AsyncResult.IsCompleted} {Now} {String.Join(".", SshOutput)}")
  21:          'Debug.Print($"#3 {IpAddr} {AsyncResult.IsCompleted} {Now} ")
  22:          SshCommand.EndExecute(AsyncResult)
  23:          Await Progress(SshCommand, StdoutSR, StderrSR, OutputLine, CTX, SshOutput, SshErrMsg, IpAddr)
  24:          'Debug.Print($"#4 {AsyncResult.IsCompleted} {Now} {String.Join(".", SshOutput)}")
  25:          'Debug.Print($"#4 {IpAddr} {AsyncResult.IsCompleted} {Now} ")
  26:      End Function
  27:   
  28:      Private Async Function Progress(ByVal SshCommand As SshCommand, ByVal StdoutSR As TextReader, ByVal StderrSR As TextReader, ByVal OutputLine As IProgress(Of SshOutputLine), ByVal CTX As CancellationToken, SshOutput As List(Of String), SshErrMsg As List(Of String), IpAddr As String) As Task
  29:          If CTX.IsCancellationRequested Then SshCommand.CancelAsync()
  30:          CTX.ThrowIfCancellationRequested()
  31:          Await OutProgress(StdoutSR, OutputLine, SshOutput, SshErrMsg, IpAddr)
  32:          Await ErrProgress(StderrSR, OutputLine, SshOutput, SshErrMsg, IpAddr)
  33:      End Function
  34:   
  35:      Private Async Function OutProgress(ByVal StdoutSR As TextReader, ByVal StdoutProgress As IProgress(Of SshOutputLine), SshOutput As List(Of String), SshErrMsg As List(Of String), IpAddr As String) As Task
  36:          Dim StdoutLine = Await StdoutSR.ReadToEndAsync()
  37:          If Not String.IsNullOrEmpty(StdoutLine) Then StdoutProgress.Report(New SshOutputLine(Line:=StdoutLine, IsErrorLine:=False, SshOutput, SshErrMsg, IpAddr))
  38:      End Function
  39:   
  40:      Private Async Function ErrProgress(ByVal StderrSR As TextReader, ByVal stderrProgress As IProgress(Of SshOutputLine), SshOutput As List(Of String), SshErrMsg As List(Of String), IpAddr As String) As Task
  41:          Dim StderrLine = Await StderrSR.ReadToEndAsync()
  42:          If Not String.IsNullOrEmpty(StderrLine) Then stderrProgress.Report(New SshOutputLine(Line:=StderrLine, IsErrorLine:=True, SshOutput, SshErrMsg, IpAddr))
  43:      End Function
  44:  End Module

Debugging is commented.



That's it, description of this engine will be continue in next pages.



Comments ( )
Link to this page: http://www.vb-net.com/SshQueueNotificationServer/Index.htm
< THANKS ME>