Sharing experience

[Puzzak]

Heyo. I don't want to fork/pull directly because what I did with the plugin changes the logic pretty severely.
There are couple of quirks (like I implemented summarization via custom streaming), but overall changes are:

• Move from the normal generateContent to generateContentStream
  • Basically this means that all interactions with model are now streamed. Instead of "send request, wait for response" loop, I implemented "send request, gather fragments and update the UI untill you receive all-done from AI Core"
• Added several features, as example a way to get model download status (also streamed) so the ui can be updated

Full disclosure, I am not good with Kotlin, so it was a long and painfull process, while I learned relevant Kotlin with the help of AI. There may be some errors or arhcitectural issues, but overall it looks okay-ish. I am willing to give what I learned back to you since if this will be helpful for you, it will benefit everyone.

My changes are in 2 files, one is the Kotlin plugin, one is the slightly reworked dart connector that communicates with it. They are attached.

This here is the Dart side of the plugin
gemini.dart.txt
And this is the plugin itself
FlutterLocalAiPlugin.kt.txt

Main changes are with initAiCore:

val status = generativeModel!!.checkStatus()
            when (status) {
                FeatureStatus.AVAILABLE -> "Available=Available=0"
                FeatureStatus.UNAVAILABLE -> "Error=AICore=Unavailable"
                FeatureStatus.DOWNLOADING -> "Download=In Progress=0"
                FeatureStatus.DOWNLOADABLE -> {
                    var downloadMessage = "Download=In Progress=0"
                    generativeModel!!.download().collect {
                        status -> when (status) {
                            is DownloadStatus.DownloadStarted ->   downloadMessage = "Download=In Progress=0"
                            is DownloadStatus.DownloadProgress ->  downloadMessage = "Download=In Progress=${status.totalBytesDownloaded}"
                            is DownloadStatus.DownloadCompleted -> downloadMessage = "Available=Download=0"
                            is DownloadStatus.DownloadFailed ->    downloadMessage = "Error=Download=${status.e.message}"
                            else ->                                downloadMessage = "Error=Download=Unknown"
                        }
                    }
                    downloadMessage
                }
                else -> "Error=Unknown=$status"
            }

The statuses are then split by = and didplayed in the app.

In the generateTextStream (new one):

return generativeModel!!.generateContentStream(request)
            .transform { chunk ->
                val newChunkText = chunk.candidates.firstOrNull()?.text ?: ""
                val finishReason = chunk.candidates.firstOrNull()?.finishReason.toString()
                fullResponse += newChunkText
                val generationTime = System.currentTimeMillis() - startTime
                val tokenCount = fullResponse.split(" ").filter { it.isNotEmpty() }.size
                emit(mapOf(
                    "text" to fullResponse,
                    "chunk" to newChunkText,
                    "generationTimeMs" to generationTime,
                    "tokenCount" to tokenCount,
                    "reason" to finishReason
                ))
            }

I am now working with chunks, which allows me to update the chat immediately after receiving the text part instead of waiting for the whole text to arrive.

In the handleGenerateTextStream I use coroutine to continuously monitor the status of each chunk and relay it to the Dart code:

coroutineScope.launch {
            try {
                withContext(Dispatchers.Main) {
                    events.success(mapOf("status" to "Loading", "response" to null, "error" to null))
                }

                generateTextStream(prompt, configMap)
                    .onEach { chunkMap ->
                        val reasonFromChunk = chunkMap["reason"] as? String
                        if (reasonFromChunk != null && reasonFromChunk != "null") {
                            lastFinishReason = reasonFromChunk
                        }
                        withContext(Dispatchers.Main) {
                            events.success(mapOf("status" to "Streaming", "response" to chunkMap, "error" to null))
                        }
                    }
                    .onCompletion {
                        withContext(Dispatchers.Main) {
                            events.success(mapOf("status" to "Done", "response" to null, "error" to null, "reason" to lastFinishReason))
                            events.endOfStream()
                        }
                    }
                    .catch { e ->
                        withContext(Dispatchers.Main) {
                            events.success(mapOf("status" to "Error", "response" to null, "error" to e.message))
                            events.endOfStream()
                        }
                    }
                    .launchIn(this)

            } catch (e: Exception) {
                withContext(Dispatchers.Main) {
                    events.success(mapOf("status" to "Error", "response" to null, "error" to e.message))
                    events.endOfStream()
                }
            }
        }

This is all sent via EventChannel and received via MethodChannel:

override fun onListen(arguments: Any?, events: EventChannel.EventSink?) {
        if (events == null) return

        val args = arguments as? Map<String, Any>
        val method = args?.get("method") as? String
        val payload = args?.get("payload") as? Map<String, Any>

        if (method == null || payload == null) {
            events.error("INVALID_ARGS", "Missing 'method' or 'payload' in EventChannel arguments", null)
            return
        }

        when (method) {
            "generateText" -> handleGenerateText(payload, events)
            "generateTextStream" -> handleGenerateTextStream(payload, events)
            else -> events.error("UNKNOWN_METHOD", "Unknown method '$method' for EventChannel", null)
        }
    }

[Puzzak]

Alright. Now with Dart.
I am sending and receiving all data via

  static const MethodChannel _methodChannel = MethodChannel('flutter_local_ai');      /// Send instructions for Kotlin part
  static const EventChannel _eventChannel = EventChannel('flutter_local_ai_events');  /// Receive updates
  EventChannel downloadChannel = EventChannel('download_channel');                    /// Status of model
  late Stream<String> statusStream;

I receive the updates and categorize them in my engine - read backend of the app's UI:

    _aiSubscription = stream.listen(
          (AiEvent event) async {
        switch (event.status) {
          case AiEventStatus.loading:
            isLoading = true;
            responseText = "";
            status = dict.value("waiting_for_AI");
            notifyListeners();
            break;

          case AiEventStatus.streaming:
            isLoading = true;
            String? finishReason = event.response?.finishReason;
            if(!(event.response?.finishReason=="null")) {
              switch(finishReason??"null"){
                case "0": if (kDebugMode) {print("Generation stopped (MAX_TOKENS): The maximum number of output tokens as specified in the request was reached.");}break;
                case "1": if (kDebugMode) {print("Generation stopped (OTHER): Generic stop reason.");}break;
                case "-100": if (kDebugMode) {print("Generation stopped (STOP): Natural stop point of the model.");}break;
                default: if (kDebugMode) {print("Generation stopped (Code ${event.response?.finishReason}): Reason for stop was not specified");}break;
              }
            }
            status = "Streaming response...";
            if (event.response != null) {
              response = event.response!;
              responseText = event.response!.text;
            }
            scrollChatlog(Duration(milliseconds: 250));
            await Future.delayed(Duration(milliseconds: 500));
            scrollChatlog(Duration(milliseconds: 250));
            break;

          case AiEventStatus.done:
            if(responseText == ""){
              if(errorRetry){
                if(event.response?.text == null){
                  isLoading = false;
                  Fluttertoast.showToast(
                      msg: "Unable to generate response.",
                      toastLength: Toast.LENGTH_SHORT,
                      fontSize: 16.0
                  );
                }else{
                  await Future.delayed(Duration(milliseconds: 500));
                  generateStream();
                }
              }else{
                isLoading = false;
                isError = true;
                status = "Error";
                responseText = event.error ?? "Unknown stream error";
              }
            }else{
              isLoading = false;
              status = "Done";
              addToContext();
              prompt.clear();
              scrollChatlog(Duration(milliseconds: 250));
            }
            break;

          case AiEventStatus.error:
            if(errorRetry){
              await Future.delayed(Duration(milliseconds: 500));
              generateStream();
            }else{
              isLoading = false;
              isError = true;
              status = "Error";
              responseText = event.error ?? "Unknown stream error";
            }
            break;
        }
        notifyListeners();
      },

There is a lot of irrelevant junk that just manages app state, please ignore it - I'm a bit too tired (5 am for me) to fix that

---

The following function ingests text and gives short summary. It is a very lame way to do it, but it works (and pretty good at that).

  Future generateTitle (String input) async {
    ignoreContext = true;
    String newTitle = "Getting description";
    await gemini.init().then((initStatus) async {
      ignoreContext = false;
      if (initStatus == null) {
        analyzeError("Initialization", "Did not get response from AICore communication attempt");
      }else{
        if (initStatus.contains("Error")) {
          analyzeError("Initialization", initStatus);
        }else{
          await gemini.generateText(
            prompt: "Task: Create a short, 3-5 word title for the user's message.\n"
                "Rules:\n"
                "1. DO NOT use full sentences.\n"
                "2. DO NOT use phrases like \"The text is about\" or \"Summary of\".\n"
                "3. Be extremely concise.\n"
                "4. The title MUST be in the same language as the input.\n"
                "Examples:\n"
                "Input: \"Hello, how are you?\"\n"
                "Title: Greeting\n\n"
                "Input: \"Привіт, як справи?\"\n"
                "Title: Привітання\n\n"
                "Input: \"Write a python script to sort a list\"\n"
                "Title: Python sorting script\n\n"
                "Input: \"Why is the sky blue?\"\n"
                "Title: Sky color explanation\n\n"
                "Input: \"I need help with my printer\"\n"
                "Title: Printer troubleshooting\n\n"
                "Input: \"sdlkfjsdf\"\n"
                "Title: Random characters\n\n"
                "Input: \"$input\"\n"
                "Title:",
            config: GenerationConfig(maxTokens: 20  , temperature: 0.7),
          ).then((title){
            newTitle = title.split('\n').first;
            newTitle = newTitle.replaceAll(RegExp(r'[*#_`]'), '').trim();
            if (newTitle.length > 40) {
              newTitle = "${newTitle.substring(0, 40)}...";
            }
          });
        }
      }
    });
    return newTitle.trim().replaceAll(".", "");
  }

And the check for AICore is looking like this (again, lots of junk, look at the structure instead):

  Future<void> checkEngine() async {
    if(modelDownloadLog.isEmpty){
      lateNetCheck();
      lateProgressCheck();
    }
    modelDownloadLog.clear();
    gemini.statusStream = gemini.downloadChannel.receiveBroadcastStream().map((dynamic event) => event.toString());
    gemini.statusStream.listen((String downloadStatus) async {
      switch (downloadStatus.split("=")[0]){
        case "Available":
          modelInfo = await gemini.getModelInfo();
          if(modelInfo["version"]==null){
            await Future.delayed(Duration(seconds: 2));
            checkEngine();
          }else{
            if(modelInfo["status"]=="Available"){
              addDownloadLog("Available=Available=0");
              endFirstLaunch();
            }else{
              if(downloadStatus.split("=")[1] == "Download"){
                if(!(modelDownloadLog[modelDownloadLog.length-1]["info"] == "waiting_network")){
                  addDownloadLog("Download=downloading_model=0");
                }
              }else{
                addDownloadLog(downloadStatus);
              }
            }
          }
          break;
        case "Download":
          if(modelDownloadLog.isEmpty){
            addDownloadLog(downloadStatus);
          }else{
            if(!modelDownloadLog[modelDownloadLog.length-1]["value"].contains("error")){
              if(int.parse(downloadStatus.split("=")[2]) > int.parse(modelDownloadLog[modelDownloadLog.length-1]["value"])){
                addDownloadLog(downloadStatus);
              }
            }
          }
          break;
        case "Error":
          addDownloadLog(downloadStatus);
          if(downloadStatus.split("=")[2].contains("1-DOWNLOAD_ERROR")){
            checkEngine();
          }else{
            Fluttertoast.showToast(
                msg: "Gemini Nano ${dict.value("unavailable")}",
                toastLength: Toast.LENGTH_SHORT,
                fontSize: 16.0
            );
          }
          break;
        default:
          addDownloadLog(downloadStatus);
      }
    },
      onError: (e) async {
        analyzeError("Received new status: ", e);
      },
      onDone: () {
        notifyListeners();
      },
    );
  }

All interactions with AICore feel junky and you need to "force" it to "behave" so it just works. There are lots of lame solutions for that, but it is a proof of concept and this will be refined later.

In any case I ecourage you to fork the GeminiLocal and tweak it if this relevant to you: https://github.com/Puzzaks/geminilocal

[Puzzak]

I know this sounds pretentious, after all your plugin gave me the idea and was the starting point - and FWIW I may very well be explaining this to you when you already discovered eveything here and this all is just stupid. But.

But I am just giving you a rundown on what I found after I did some research based on your plugin, and looking at the "under construction" in the "streamed generation for Android" cell in the readme, I am hoping this may help you figure out how this may work for Android.

I hope this is not offensive, and if it is - I am deeply sorry, I'm always grateful for your plugin and what it allowed me to do, and I wish you much luck in your work.

You helped me a lot, and your plugin will be always linked in my app. Even the Gemini Nano by default has the link to your repo in the system prompt.

Thank you again.

[kekko7072]

Hi @Puzzak thank you very much for the experience.

Can you test the apk of the example and see if it works as i'm having problem to get and android device with those capabilities and the Firebase Test Lab with real device doesn't work. I opened an issue but still waiting for the response.